Ted is writing thingshttps://desfontain.es/privacy/2023-12-28T00:00:00+01:00Choosing things privately with the exponential mechanism2023-10-09T00:00:00+02:002023-10-09T00:00:00+02:00Damien Desfontainestag:desfontain.es,2023-10-09:/privacy/choosing-things-privately.html<p>A simple introduction to an essential building block for differential privacy: how to select a value among many.</p><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p></small></p>
<hr>
<p><span class='lettrine'>T</span><strong>he</strong> goal of most differential privacy
mechanisms is to publish <em>statistics</em>: numbers giving some information about
groups of people. But to build more complex DP mechanisms, we sometimes need a
different kind of building block. In this article, we won't be adding noise to
numbers. Instead, we'll make a <em>choice</em> among multiple options, in a
differentially private way.</p>
<h1 id="a-simple-example">A simple example</h1>
<p>Let's say that we're designing a poll to pick the best science-fiction books
published in 2020. First, we'll collect a big list of books published that year.
Then, we ask people to select the books they liked. Each person can pick as many
books as they want, and we want to select the book with most votes. If there is
a tie, we select the winner randomly.</p>
<p>How do we publish this in a privacy-preserving way? Let's take a look at the
books with the most votes in our voting results.</p>
<p><img alt="A bar chart showing the number of votes on 6 books; the x-axis ends in a
dotted line, suggesting some books are not being shown. "Network Effect" has 50
votes, "Hench" and "The Hidden Girl" have 49 votes, "The Relentless Moon" has 47
votes, "Axiom's End" and "Riot Baby" have 46 votes." src="https://desfontain.es/privacy/images/book-votes.svg"></p>
<p>We can't simply publish the true answer without randomness. Otherwise, we would
be publishing the most popular book (here, <em>Network Effect</em>) 100% of the time.
But imagine that we add a single person, who only votes for one of the
second-most popular books.</p>
<p><img alt="The same bar chart as before, except "The Hidden Girl" now has 50 votes, and
its bar is a slightly darker shade of blue." src="https://desfontain.es/privacy/images/book-votes-tie.svg"></p>
<p>In that case, we would want to release either one or the other with equal
probability. This is a violation of differential privacy: 50% of the time, we
publish <em>The Hidden Girl</em>, and this couldn't happen without this added user!</p>
<p><img alt="A bar chart as before with the same books in the x-axis; now the y-axis is
labeled "Probability of being selected as winner". In light blue, legended
"Original data", there is only one bar, at probability 1, for "Network Effect".
In dark blue, legended "With one added vote", there are two bars at probability
0.5: one for "Network Effect" and one for "The Hidden Girl". The bars for "The
Hidden Girls" are labeled 0 and 0.5, in red, and circled; red text above the
circle says "DP violation!"." src="https://desfontain.es/privacy/images/book-selection-dp-violation.svg"></p>
<p>To avoid this, we need to add more randomness in our process. How do we do that
exactly?</p>
<p>Well, we already know how to publish <a href="differential-privacy-in-practice#many-things">histograms</a> with differential privacy:
we add well-calibrated Laplace noise to each of the statistics. So why don't we
just do that? If we make the entire histogram private, we can release all of it.
In particular, we can see which book has the highest noisy vote count, and
declare it the winner.</p>
<p>But we have a problem here. The noise has to be scaled by the number of
statistics that each user can contribute to. In our setting, a particularly
enthusiastic user could vote for <em>all the books</em>. If our list has 10,000 books
to choose from, then we have to multiply the noise scale by 10,000. This seems…
not great. And it also feels unnecessary: we don't <em>want</em> to release the entire
histogram, we only want to pick a winner. Could we use that fact to inject less
noise into the process?</p>
<p>Let's try to think about what a good strategy would look like. Say we are using
<span class="math">\(\varepsilon=\ln(2)\)</span>. Let's look at the votes for the most popular books again.</p>
<p><img alt="The same bar chart as the first one, showing the number of votes on 6 books.
The x-axis ends in a dotted line, suggesting some books are not being shown.
"Network Effect" has 50 votes, "Hench" and "The Hidden Girl" have 49 votes, "The
Relentless Moon" has 47 votes, "Axiom's End" and "Riot Baby" have 46
votes." src="https://desfontain.es/privacy/images/book-votes.svg"></p>
<p>The true winner is <em>Network Effect</em>. So we want to select this true answer with
some probability, hopefully as high as possible.</p>
<p><img alt="A bar chart showing the probability distribution of the book selected as a
winner. It has a single bar for "Network Effect", all other bars are empty and
replaced by question marks." src="https://desfontain.es/privacy/images/book-selection-single-probability.svg"></p>
<p>Now, what is the probability of selecting one of the second-best choices? They
both have one fewer vote than the winner. They're <em>not</em> the correct answer, so
we want to select them with as small a probability as possible. But we're also
constrained by our differential privacy guarantee.</p>
<p>Imagine that we add a single new person to the data, and they vote for <em>Hench</em>.
Then, we should be selecting <em>Network Effect</em> and <em>Hench</em> with equal
probability.</p>
<p><img alt="The same bar chart as earlier, with an additional grey & dotted-line bar for
"Hench", of equal height to the one for "Network
Effect"." src="https://desfontain.es/privacy/images/book-selection-hypothetical.svg"></p>
<p>With DP, we must select <em>Hench</em> with similar probability as in this hypothetical
scenario. How similar? We chose <span class="math">\(e^\varepsilon=2\)</span>, so there can be a factor of
at most 2 between these probabilities. We want it to be as small as possible, so
let's make it exactly half.</p>
<p><img alt="The same bar chart as earlier, with an additional blue bar for "Hench", whose
height is half that of the grey bar. A red arrow labeled "Selection probability
divided by e^ε = 2" goes from the grey bar to the blue
bar." src="https://desfontain.es/privacy/images/book-selection-two-probabilities.svg"></p>
<p>The exact same reasoning holds for <em>The Hidden Girl</em>, which has as many votes as
<em>Hench</em>.</p>
<p><img alt="A bar chart showing the probabilities of the three books with most votes.
"Network effect" has the biggest bar, "Hench" and "The Hidden Girl" each have a
bar whose height is half, all other books have a question
mark." src="https://desfontain.es/privacy/images/book-selection-three-probabilities.svg"></p>
<p>What about one of the books that is a little further away from the winner? <em>The
Relentless Moon</em>, for example, is 3 votes short: we would need three more votes
to get to the winning probability. If we can add one person at a time, we need
<em>three steps</em> to get there.</p>
<p><img alt="The number of votes for each book; with three arrows on top of the bar for
"The Relentless Moon", labeled "3 more votes needed to reach the
winner"" src="https://desfontain.es/privacy/images/book-votes-three-steps.svg"></p>
<p>And each time we add one person, we have to respect the differential privacy
constraint: we can at most double the probability of selecting <em>The Relentless
Moon</em>. To arrive there after these three steps, we need to start from at least
<span class="math">\(\left(\frac{1}{2}\right)^3=1/8\)</span> of the maximum probability.</p>
<p><img alt="A bar chart showing the probabilities of the four books with most votes.
"Network effect" has the biggest bar, "Hench" and "The Hidden Girl" each have a
bar whose height is half, "The Relentless Moon" has a bar whose height is 8
times smaller, the other two books have a question
mark." src="https://desfontain.es/privacy/images/book-selection-four-probabilities.svg"></p>
<p>We can repeat this idea and draw our full probability distribution. For each
book, if it's <span class="math">\(k\)</span> votes short of the true winner, its selection probability
should be <span class="math">\(1/2^k\)</span> of the true winner.</p>
<p><img alt="A bar chart showing the probabilities of all the books displayed on the chart.
"Network effect" has the biggest bar, "Hench" and "The Hidden Girl" each have a
bar whose height is half, "The Relentless Moon" has a bar whose height is 8
times smaller, "Axiom's End" and "Riot Baby" each have a bar whose height is 16
times smaller." src="https://desfontain.es/privacy/images/book-selection-all-probabilities.svg"></p>
<p>Achieving this for an arbitrary <span class="math">\(\varepsilon\)</span> is straightforward: the
probability of selecting a book <span class="math">\(i\)</span> with <span class="math">\(k_i\)</span> votes should be proportional to
<span class="math">\(\exp\left(\varepsilon \cdot k_i\right)\)</span>. After normalizing to make the
probabilities sum to <span class="math">\(1\)</span>, we get the following formula:</p>
<div class="math">$$
\mathbb{P}\left[\text{We choose book }i\right] =
\frac{\exp\left(\varepsilon \cdot k_i\right)}{\sum_i \exp\left(\varepsilon \cdot k_i\right)}.
$$</div>
<p>We call this DP procedure the <em>exponential mechanism</em>.</p>
<h1 id="a-generic-statement-and-a-simple-optimization">A generic statement and a simple optimization</h1>
<p>The example above is very simple: each book is simply associated to the nubmer
of votes it received. But the exponential mechanism is much more generic, and we
can use it in more complex settings. Let's say we have a database <span class="math">\(D\)</span>, and we
have to choose between many items <span class="math">\(O_1\)</span>, <span class="math">\(O_2\)</span>, and so on. We assume that each
item <span class="math">\(O_i\)</span> has a <em>score</em> <span class="math">\(s_i(D)\)</span>, which depends on the dataset. Let <span class="math">\(\Delta\)</span> be
the <em>sensitivity</em> of the scoring function: the maximum change to <span class="math">\(s_i(D)\)</span> when
one person is added to (or removed from) <span class="math">\(D\)</span>, for all <span class="math">\(i\)</span>. Then the exponential
mechanism <span class="math">\(\mathcal{M}\)</span> is defined as:</p>
<div class="math">$$
\mathbb{P}\left[\mathcal{M}(D) = O_i\right] =
\frac{\exp\left(\varepsilon\cdot\frac{s_i(D)}{2\Delta}\right)}{\sum_i \exp\left(\varepsilon\cdot\frac{s_i(D)}{2\Delta}\right)}.
$$</div>
<p>Proving that it satisfies ε-DP is very easy — the proof in the <a href="http://kunaltalwar.org/papers/expmech.pdf">original paper</a>
is just 3 lines long! Try to come up with it by yourself. Or you can also click
here: <button id="toggleProof1"></button></p>
<div id="proof1" style="display: none; border-left: double; padding-left: 10px">
<p>If you add or remove a single person from <span class="math">\(D\)</span>, the numerator of the fraction
above changes by at most a factor of <span class="math">\(e^{\varepsilon/2}\)</span>. Each term in the
denominator does the same, so the denominator also changes at most by the same
factor. So the entire probability changes by a factor of at most
<span class="math">\(e^\varepsilon\)</span>, which is exactly what we want to satisfy <span class="math">\(\varepsilon\)</span>-DP.</p>
</div>
<p>In our example above, the <span class="math">\(O_i\)</span> are the books, and the score of each book is its
number of votes. Adding or removing one person modifies the scores by at most
one, so <span class="math">\(\Delta=1\)</span>, and we get the same…</p>
<p>Wait a second. We're not getting the same thing! There's a factor of <span class="math">\(2\)</span> in the
generic formula that we didn't have in our voting example. So if we were using
the generic formula, we would get worse utility: the probability of selecting
the winner would be smaller. Can we get rid of this multiplicative factor?</p>
<p>The answer is yes, because our scores are <em>monotonic</em>: if we add a user, they
will <em>all</em> get larger. If we remove one, they will all get smaller. That's a
common special case, and in that case, you can remove the <span class="math">\(2\)</span> factor:</p>
<div class="math">$$
\mathbb{P}\left[\mathcal{M}(D) = O_i\right] =
\frac{\exp\left(\varepsilon\cdot\frac{s_i(D)}{\Delta}\right)}{\sum_i \exp\left(\varepsilon\cdot\frac{s_i(D)}{\Delta}\right)}.
$$</div>
<p>Again, this is quite easy to prove, especially if you've understood the previous
proof. <button id="toggleProof2"></button></p>
<div id="proof2" style="display: none; border-left: double; padding-left: 10px">
<p>If you add or remove a single person from <span class="math">\(D\)</span>, the numerator of the fraction
above changes by at most a factor of <span class="math">\(e^{\varepsilon}\)</span>. Each term in the
denominator will do the same, <em>in the same direction</em>: it's impossible for the
numerator to increase and for the denominator to decrease. So the maximum
multiplicative change to the probability is <span class="math">\(e^\varepsilon\)</span>.</p>
</div>
<p>Note that here, we assumed that we're protecting the addition or removal of a
single person in the dataset. If we want to protect <em>any change</em> in a single
person's votes instead, the privacy analysis changes: someone could add one vote
to a book and remove a vote to another. The scores would no longer be monotonic,
and we would need to pay the <span class="math">\(2\)</span> factor in our formula.</p>
<h1 id="more-results">More results?!</h1>
<p>The exponential mechanism is a central building block in differential privacy.
It's been studied from many different angles, so there is a lot to say about it.
This blog post is long enough already, but here are a few ✨ selected facts ✨.
Follow the links if you'd like to learn more!</p>
<ul>
<li>The exponential mechanism can be <a href="https://timvieira.github.io/blog/post/2014/07/31/gumbel-max-trick/">implemented</a> in a simple way:
add noise from a <a href="https://en.wikipedia.org/wiki/Gumbel_distribution">Gumbel distribution</a> to each score, and choose
the item with the highest noisy score.</li>
<li>Its privacy guarantees can be finely analyzed using a notion called <a href="https://differentialprivacy.org/exponential-mechanism-bounded-range/">bounded
range</a>. This allows you to prove that an exponential mechanism
calibrated for <span class="math">\(\varepsilon\)</span>-DP also satisfies <span class="math">\(\rho\)</span>-<a href="renyi-dp-zero-concentrated-dp.html">zCDP</a> with
<span class="math">\(\rho=\frac{1}{8}\varepsilon^2\)</span>: a lot better than the typical conversion of
<span class="math">\(\rho=\frac{1}{2}\varepsilon^2\)</span>.</li>
<li>Using the exponential mechanism several times? Don't use regular composition
theorems! Instead, using the special structure of this mechanism can get you
<a href="https://arxiv.org/abs/1909.13830">tighter results</a>.</li>
<li>You can do better than the exponential mechanism, with a mechanism called
<em>Permute-and-Flip</em>. Its <a href="https://arxiv.org/abs/2010.12603">original definition</a> is somewhat complicated,
but people found a <a href="https://arxiv.org/abs/2105.07260">nice characterization</a> afterwards: add noise
from a <a href="https://en.wikipedia.org/wiki/Geometric_distribution">geometric distribution</a> to each score, and pick the highest
noisy score.</li>
<li>However, the exponential mechanism still retains one advantage: it can also be
used when the space of possible choices is <em>continuous</em>, like "every real
number between 0 and 1"<sup id="fnref:continuous"><a class="footnote-ref" href="#fn:continuous">1</a></sup>. This is very useful, for example to
compute the median of values in a dataset.</li>
</ul>
<hr>
<p><small>
I'm thankful to Daniel Simmons-Marengo and PeoriaBummer for helpful feedback on this post.
</small></p>
<script type="text/javascript">
var defaultButton = 'Show me the proof';
var button1 = document.getElementById('toggleProof1');
button1.innerHTML = defaultButton
button1.addEventListener('click', function (event) {
button1.innerHTML = button1.innerHTML == defaultButton ? 'Hide the proof' : defaultButton;
proof1 = document.getElementById('proof1');
proof1.style.display = proof1.style.display == 'none' ? 'block' : 'none';
});
var button2 = document.getElementById('toggleProof2');
button2.innerHTML = defaultButton
button2.addEventListener('click', function (event) {
button2.innerHTML = button2.innerHTML == defaultButton ? 'Hide the proof' : defaultButton;
proof2 = document.getElementById('proof2');
proof2.style.display = proof2.style.display == 'none' ? 'block' : 'none';
});
</script>
<div class="footnote">
<hr>
<ol>
<li id="fn:continuous">
<p>In that case, the <em>density</em> of the probability distribution on
<span class="math">\(x\)</span> must be proportional to
<span class="math">\(\exp\left(\varepsilon\cdot\frac{s_x(D)}{2\Delta}\right)\)</span>. The normalization
factor (in the denominator) also becomes an integral instead of a discrete
sum. <a class="footnote-backref" href="#fnref:continuous" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Mapping privacy-enhancing technologies to your use cases2023-05-04T00:00:00+02:002023-09-23T00:00:00+02:00Damien Desfontainestag:desfontain.es,2023-05-04:/privacy/privacy-enhancing-technologies.html<p>A guide listing common privacy-enhancing technologies, and delineating between which problem each one solves.</p><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p> This article was first published on the <a href="https://www.tmlt.io/resources/mapping-privacy-enhancing-technologies-to-your-use-cases">Tumult Labs blog</a>; its
copyright is owned by Tumult Labs.</p>
<p></small></p>
<hr>
<p>Say you're working on a new project involving sensitive data — for example,
adding a new feature to a healthcare app. This feature is bringing new privacy
concerns that you're trying to grapple with. Maybe your lawyers aren’t feeling
great about the compliance story of the app you're building. Maybe you want to
make strong statements to users of the feature, about how you will handle their
data. Maybe you’re afraid that sensitive user data might leak in unexpected
ways. You’ve been hearing about advances in privacy technologies, and you
wonder: should I look into one of those to see if it could solve my problem?</p>
<p>You've come to the right place. In this blog post, we'll walk you through a few
key data handling use cases, each involving significant privacy challenges.
We'll then map various privacy-enhancing technologies (PETs) to those use cases.
Spoiler alert, the overall map of use cases and PETs will look like this:</p>
<p><center>
<img alt="A diagram describing multiple data operations, and the associated
privacy-enhancing technologies. First, multiple people have arrows pointing to a
database, this is labeled "Collecting data privately: secure aggregation, local
differential privacy". Multiple databases have arrows pointing towards the same
place, this is labeled "Joining data privately: multi-party computation,
confidential computing". These arrows are pointing towards a graphical
representation of a computation, this is labeled "Computing on data privately:
homomorphic encryption, confidential computing". Finally, an arrow goes from
this visual computation towards a stylized graph; this is labeled "Sharing data
privately: differential privacy"." src="https://desfontain.es/privacy/images/pets-diagram.png">
</center></p>
<p>To better understand these challenges, we will make the adversarial model
explicit as we discuss each use case. This means answering two questions:</p>
<ul>
<li>Who has access to the raw, privacy-sensitive data?</li>
<li>Who are we protecting against; who must not be able to access the raw data?</li>
</ul>
<p>In each diagram, we will label the entities with access to the data with a ✅,
and the adversaries with a ❌.</p>
<p>Let’s go through each of these categories of use cases one by one.</p>
<h1 id="collecting-data-privately">Collecting data privately</h1>
<p>For this use case, your goal is to <em>collect</em> data from individual users of your
app. For example, let's say that you want to measure some metric related to
health information among your user base. But there's a catch: you don’t want to
collect personal data. Instead, you want to be able to tell your users: "I am
not collecting data about you — I am only learning information about large
groups of users."</p>
<p><center>
<img alt="A diagram representing private data collection. Four smiley faces representing
users are on the left, and each has an arrow pointing to a database icon on the
right. Green check marks are next to each smiley face, and a "forbidden" sign is
next to the database icon. The diagram is labeled "Collecting data privately:
secure aggregation, local differential
privacy"" src="https://desfontain.es/privacy/images/collecting-data-privately.png">
</center></p>
<p>The adversarial model is as follows.</p>
<ul>
<li>Only individual users have access to their own raw data.</li>
<li>You — the organization collecting the data — must not be able to see
individual data points.</li>
</ul>
<p>Note that in the diagram above, each user has access to their own data, but
presumably not the data from other users.</p>
<p>Can you still learn something about aggregate user behavior in this context?
Perhaps surprisingly, the answer is yes! There are two main privacy technologies
that can address this use case.</p>
<ul>
<li><em>Secure aggregation</em><sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup> consists in hiding each individual value using
cryptographic techniques. These encrypted data points are then combined to
compute the aggregate result.</li>
<li><em><a href="local-global-differential-privacy.html">Local differential privacy</a></em> consists in adding random noise to
each individual data point. This noise hides the data of each person… but
combining many data points can still reveal larger trends.</li>
</ul>
<p>Both technologies can work together, and complement each other well. Local
differential privacy provides formal guarantees on the output, at the heavy cost
in accuracy. But combining it with secure aggregation can avoid most of this
accuracy cost, and boost utility while preserving strong guarantees.</p>
<p><em>Federated learning</em> is a common use case for these techniques. With this
machine learning technique, model training happens on each user’s device. This
can be better for privacy than sending the raw data to a central server… but
model upgrades from each user can still leak sensitive information! Using secure
aggregation and/or local differential privacy mitigates this risk.</p>
<h1 id="computing-on-data-privately">Computing on data privately</h1>
<p>For this use case, your goal is to have a partner <em>run computations</em> on your
sensitive data, but hide the data from this partner. For example, in our
healthcare app story, let’s say you collected some sensitive data through the
app. A partner company has built a prediction model that you want to use on this
data. You want them to run their model on your data, but you don’t want them to
be able to access your data directly.</p>
<p><center>
<img alt="A diagram representing private data computation. A database icon is on the
left, and an arrow goes from it to an icon representing a computation, on the
right. A green check mark is under the database icon, and a "forbidden" sign is
below the computation sign. The diagram is labeled "Computing on data privately:
homomorphix encryption, confidential
computing"." src="https://desfontain.es/privacy/images/computing-on-data-privately.png">
</center></p>
<p>The adversarial model is as follows.</p>
<ul>
<li>You – the institution collecting the data – have access to the sensitive data.</li>
<li>The organization performing the computation must not be able to access this data.</li>
</ul>
<p>Two main technologies address this use case.</p>
<ul>
<li><em>Homomorphic encryption</em> consists in encrypting the data before performing the
computation. The organization must adapt its computation to work on encrypted
data. Then, they send you back the result in encrypted form, and you can
decrypt it to see the result.</li>
<li><em>Confidential computing</em><sup id="fnref2:2"><a class="footnote-ref" href="#fn:2">2</a></sup> is a hardware-based approach to encrypt data
while in-use. It can be combined with remote attestation: this technique
allows you to verify that only the code that you have approved is running on
your data.</li>
</ul>
<p>The guarantee offered by homomorphic encryption is stronger: you do not need to
trust that the hardware is correctly secured. However, these stronger guarantees
come at a cost: homomorphic encryption often has a very large performance
overhead.</p>
<h1 id="joining-data-privately">Joining data privately</h1>
<p>For this use case, your goal is to <em>combine</em> your data with the data from other
organizations. For example, in our healthcare app, you might want to count how
many of your users also use another app, made by a different company. Or you
want to measure correlations between metrics in both apps. But like before, you
don’t want anybody else accessing your data directly. And you don’t want to see
the data from the other organizations, either!</p>
<p><center>
<img alt="A diagram representing private data joins. Three database icons are on the
left, the first one being larger than others. Arrows point from each database
icon to a "computation" icon. A green check mark is under the first bigger
database icon; forbidden signs are below the other two databases, and the
computation icon. The diagram is labeled "Joining data privately: multi-party
computation, confidential
computing"." src="https://desfontain.es/privacy/images/joining-data-privately.png">
</center></p>
<p>The adversarial model is as follows.</p>
<ul>
<li>You have access to your sensitive data (and only yours).</li>
<li>Other organizations must not be able to access it. The platform running the
computation (if any) must also not be able to access it.</li>
</ul>
<p>What about the output of the computation — who can access it? It depends. In
some cases, all participating organizations can access the results. In others,
only some organizations can see them.</p>
<p>There are two main technologies that address this use case.</p>
<ul>
<li><em>Secure multi-party computation</em> consists in each participant first encrypting
their own data. Then, participants use a cryptographic protocol to compute the
metric of interest.</li>
<li><em>Confidential computing</em><sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup> uses hardware modules to encrypt data while in-use.
Like before, it works best when combined with remote attestation: then, every
participant can verify that only approved code is running on their data.</li>
</ul>
<p>Note that these techniques are sometimes not enough to protect the original
data: the result of the computation can in itself leak something sensitive about
the data! And this is the perfect transition for our next use case…</p>
<h1 id="sharing-data-privately">Sharing data privately</h1>
<p>Finally, for this use case, your goal is to analyze your data, and share some
insights about it. Here, sharing can mean very different things.</p>
<ul>
<li><em>Internal sharing</em>: employees from another department of your organization
might want to use your app metrics to inform the design of a different
product. However, sharing personal data would require explicit consent in your
privacy policy: your compliance story requires that you correctly anonymize
metrics, even for internal use.</li>
<li><em>External sharing</em>: researchers from a partner university might want to use
data from your app for a scientific study. Your goal is to share insights with
them, without allowing them to see individual information.</li>
<li><em>Publication</em>: you might want to show some aggregated metrics in the app
itself as part of a feature. In this case, all users of your app can see these
metrics: it’s critical that they don't inadvertently reveal private
information. </li>
</ul>
<p>Removing identifiers is, of course, <a href="trustworthy-anonymization.html">not enough to mitigate privacy
risk</a>. How do you enable such use cases without revealing individual
information?</p>
<p><center>
<img alt="A diagram representing differentially private sharing. A database icon is on
the left, an arrow points to a stylized bar chart and line chart. A green check
mark is below the database icon, a "forbidden" sign is below the chart icon. The
diagram is labeled "Sharing data privately: differential
privacy"." src="https://desfontain.es/privacy/images/joining-data-privately.png">
</center></p>
<p>The adversarial model is as follows.</p>
<ul>
<li>You have access to the sensitive raw data.</li>
<li>People who can see the shared data cannot use it to learn information about
individuals.</li>
</ul>
<p>There is one main technology that addresses this use case. If you're reading
this <a href="friendly-intro-to-differential-privacy.html">blog post series</a>, you certainly know what it is: differential
privacy<sup id="fnref:3"><a class="footnote-ref" href="#fn:3">3</a></sup>. It adds statistical noise to aggregated information and provides
strong privacy guarantees. You can use differential privacy for different kinds
of data releases:</p>
<ul>
<li>statistics or other aggregated analyses on the original dataset;</li>
<li>machine learning models trained on the sensitive data;</li>
<li>or synthetic data, which has the same format as the original data.</li>
</ul>
<p>This is what me and my colleagues at <a href="https://tmlt.io">Tumult Labs</a> are focusing on,
building <a href="https://tmlt.dev">open-source software</a> and providing solutions to tailored to
our customer's needs. <a href="http://tmlt.io/connect">Reach out</a> if that sounds like something you
could use!</p>
<h1 id="final-comments">Final comments</h1>
<p>Handling sensitive data comes with many challenges. In this blog post, I've
listed a few major use cases, and the privacy technologies that address them. I
omitted some other privacy-enhancing technologies, for two distinct reasons.</p>
<ul>
<li>Some approaches for the use cases we’ve seen do not provide any robust privacy
guarantee. For example, some providers address the "joining data privately"
use case without provable guarantees: instead, they simply present themselves
as trusted third-parties. The situation is similar for the "sharing data
privately" use case: some providers focus on ad hoc anonymization techniques.
These do not make it possible to formally quantify privacy risk, and often
<a href="trustworthy-anonymization.html">fail in practice</a>.</li>
<li>Some technologies address more niche or infrequent use cases. For example,
<em>zero-knowledge proofs</em> are mainly useful in cryptocurrency/blockchain
applications. <em>Private information retrieval</em> can make a database accessible
to clients, without being able to learn which part of the data these clients
are querying. And there are others: privacy technology is a big space, with
constant innovation.</li>
</ul>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>Sometimes called <em>federated analytics</em>. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>The term "confidential computing" has several synonyms and related
concepts.</p>
<ul>
<li><em>Trusted execution environments</em> refer to the hardware modules used in
confidential computing.</li>
<li><em>Trusted computing</em> uses the same kind of hardware modules as confidential
computing. But in trusted computing, the context is different: end users,
rather than organizations, are running the hardware module on their
devices. Digital rights management is a common use case for this setting.</li>
<li><em>Data cleanrooms</em> is a more generic term for confidential computing: it
also includes more ad hoc solutions that do not use trusted hardware
modules.</li>
</ul>
<p><a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a><a class="footnote-backref" href="#fnref2:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>Which, here, is used as a shortcut for <a href="local-global-differential-privacy.html"><em>central differential
privacy</em></a>. This isn't the most explicit, but is often done in
practice. <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>What anonymization techniques can you trust?2023-03-10T00:00:00+01:002023-03-10T00:00:00+01:00Damien Desfontainestag:desfontain.es,2023-03-10:/privacy/trustworthy-anonymization.html<p>An overview of legacy techniques used to anonymize data, how they fail, and what we can learn from these failures.</p><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p> This article was first published on the <a href="https://www.tmlt.io/resources/what-anonymization-techniques-can-you-trust">Tumult Labs blog</a>; its
copyright is owned by Tumult Labs.</p>
<p></small></p>
<hr>
<p>Let's say that we have some sensitive data, for example about people visiting a
hospital. We would like to share it with a partner in an anonymous way: the goal
is to make sure that the released data does not reveal anything about any one
individual. What techniques are available for this use case?</p>
<h1 id="randomize-identifiers">Randomize identifiers</h1>
<p>Obviously, if we leave names, or public user identifiers in our data (like
people's telephone numbers or email addresses), then that's not going to be
anonymous. So here is a first idea: let's hide this information! By replacing
e.g. names with random numbers, identities are no longer obvious. This is called
<em>pseudonymization</em> (or sometimes <em>tokenization</em>): identifiers are replaced with
<em>pseudonyms</em> (or <em>tokens</em>). These pseudonyms are consistent: the same original
identity is always replaced by the same pseudonym.</p>
<p><center>
<img alt="A diagram showing the process of randomizing identifiers. A table has headers
Name, Date of birth, ZIP code, Visit date, and Visit reason; a single row has
values Taylor Lewis, 1987, 14217, 2022-03-30, and Flu. An arrow goes from this
table to another table with the same columns headers and values, except the Name
column has been replaced with Pseudonym, and the pseudonym value is a string of
numbers." src="https://desfontain.es/privacy/images/randomizing-identifiers.png">
</center></p>
<p>Unfortunately, "no longer obvious" is <em>very</em> different from "impossible to
figure out". Randomizing identifiers often fails to protect the privacy of the
people in the data. This can be because this randomization process itself is
insecure. A good example is the New York taxi database data. The randomization
process was done in a naive way… and this allowed researchers to
<a href="https://www.theguardian.com/technology/2014/jun/27/new-york-taxi-details-anonymised-data-researchers-warn">reverse-engineer license plates</a> from pseudonyms.</p>
<p>But there is a more fundamental reason why such schemes are unsafe: it's
impossible to know for sure what can be used to re-identify someone. Direct
identifiers are not the only thing that can be used to find out someone's
identity. A famous example is the release of AOL search queries. AOL data
scientists randomized all the identifiers. But the data itself was problematic:
what you search for reveals a lot about you! <a href="https://www.nytimes.com/2006/08/09/technology/09aol.html">It only took a few days for
journalists to reidentify people</a>, using only their search queries.</p>
<p>Even worse, otherwise-innocuous data can become identifying when combined with
additional information. The Netflix Prize dataset provides a striking example of
this fact. Netflix published pseudonymized data containing only movie ratings.
These do not seem identifying… and yet, researchers could <a href="https://www.wired.com/2007/12/why-anonymous-data-sometimes-isnt/">combine them with
public reviews</a> and recover users' identities. </p>
<h1 id="remove-identifiers-altogether">Remove identifiers altogether</h1>
<p>If pseudonymization doesn't work, what about <em>de-identification</em>? Instead of
replacing direct identifiers with random numbers, we could redact them
altogether. This technique, sometimes called <em>masking</em>, is very common.</p>
<p><center>
<img alt="The same diagram as previously, except the Name column is also present in the
second table, but the value for this column has been replaced by
"(REDACTED)"." src="https://desfontain.es/privacy/images/removing-identifiers.png">
</center> </p>
<p>Unfortunately, masking provides little extra protection. The previous problem
still applies: how can we know what information to redact and what to keep? Time
and time again, data owners underestimate the reidentifiability of their data.</p>
<p>The Massachusetts state government gave us a first example of this phenomenon.
In the 1990s, they released medical data about hospital visits, with names
redacted. But this patient data contained key demographic information: ZIP
codes, dates of birth, and sex. And these are enough to identify a large
fraction of the population! <a href="https://arstechnica.com/tech-policy/2009/09/your-secrets-live-online-in-databases-of-ruin/">Including the then-governor of
Massachusetts…</a> More than a little embarrassing. With more
demographic attributes, reidentification risk skyrockets to up to
<a href="https://techcrunch.com/2019/07/24/researchers-spotlight-the-lie-of-anonymous-data/">99.98%</a>.</p>
<blockquote>
<p>"Isn't this sort of obvious?" – <a href="https://www.wired.com/2007/12/why-anonymous-data-sometimes-isnt/">Wired, 2007</a></p>
</blockquote>
<p>A lot of data turns out to be identifying, besides demographic information.
<a href="https://archive.nytimes.com/bits.blogs.nytimes.com/2015/01/29/with-a-few-bits-of-data-researchers-identify-anonymous-people/">Credit card metadata</a>, <a href="http://edition.cnn.com/2013/03/26/tech/mobile/mobile-gps-privacy-study/">location information</a>, or <a href="https://www.sciencenews.org/article/ai-identify-anonymous-data-phone-neural-network">social
interactions</a> can be just as revealing. The problem is profound: there
is no way to know what a malicious person might use to reidentify records in our
data. The only safe choice is to redact all the data, which is not very useful.</p>
<h1 id="apply-rule-based-techniques">Apply rule-based techniques</h1>
<p>Since simpler techniques fail, we could try more complicated heuristics. Many of
them appear in the literature, and are still in use today:</p>
<ul>
<li>adding some random perturbation to individual values;</li>
<li>making some attributes less granular;</li>
<li>suppressing records with rare values;</li>
<li>and a myriad of others.</li>
</ul>
<p>These techniques might seem less naive, but they still don't provide a robust
guarantee.</p>
<p><center> <img alt="The same diagram as previously, except that in addition to redacting
the Name column, the Date of birth column has been replaced with 1987-04-17 (a
slightly different date than the original one), and the ZIP code column has been
replaced with 14***." src="https://desfontain.es/privacy/images/rule-based-techniques.png">
</center> 

The most striking example is probably the release of <a href="https://www.zdnet.com/article/re-identification-possible-with-australian-de-identified-medicare-and-pbs-open-data/">Medicare records in
Australia</a>. To limit reidentifiability risk, the organization went one step
further than simply removing identifiers: they perturbed some attributes, and
randomly shifted all dates by a random amount. The only data left was medical
information, which didn't seem like something an attacker could know about! But
for famous people, like politicians or athletes, some of this data can be
public! This allows reidentification, and retrieval of additional private
medical data.</p>
<p>Even privacy notions from the scientific literature can fail to protect
sensitive data. The first and most famous of these definitions is probably
<a href="k-anonymity.html">k-anonymity</a>. Its intuition seems convincing: each individual is
"hidden in a group" of other people with the same characteristics. Sadly,
despite this intuition, k-anonymity fails at providing a good level of
protection: <a href="https://techxplore.com/news/2022-10-kind-downcoding-flaws-anonymizing.html">downcoding attacks</a> succeed at reidentifying people in
data releases.</p>
<h1 id="aggregate-the-data">Aggregate the data</h1>
<p>It seems like trying to look at each individual record to try to find out what
to redact or randomize doesn't work. What if we aggregate multiple records
together, instead? Surely releasing statistics across many people should be
safe?</p>
<p><center>
<img alt="A diagram similar to before, but with multiple rows in both tables. In the
first table, each row represents different people, and three rows contain fake
data about three distinct people. The second table has columns Visit date, Visit
reason, Count, and Average age, and multiple rows with different fake values,
representing statistics for different visit dates and
reasons." src="https://desfontain.es/privacy/images/aggregating-data.png">
</center></p>
<p>Sadly, this is still not the case: there are multiple ways that individual
information can be retrieved from aggregated data. One of these ways uses the
correlations present in the data. Consider a dataset counting how many people
were in specific areas over time. This doesn't seem very identifying… Except
human mobility data tends to be predictable: people travel approximately in the
same direction between two points. This creates correlations, which attackers
can then exploit: researchers managed to <a href="https://blog.acolyer.org/2017/05/15/trajectory-recovery-from-ash-user-privacy-is-not-preserved-in-aggregated-mobility-data/">retrieve individual
trajectories</a> from such an aggregated dataset.</p>
<p>And there is another complication: it is often possible to combine multiple
statistics and retrieve individual records. This technique is called a
reconstruction attack. The most prominent example was done by the <a href="us-census-reconstruction-attack.html">U.S. Census
on the 2010 Decennial Census</a>. The results speak for themselves! Worse
still, reconstruction attacks are <a href="https://www.pnas.org/doi/10.1073/pnas.2218605120">improving over time</a>… so they
could become even more of a risk in the future.</p>
<h1 id="what-do-these-attacks-have-in-common">What do these attacks have in common?</h1>
<p>Let's take a step back and look at all these failures of bad anonymization
techniques. Are there some themes we can discern?</p>
<ul>
<li><strong>Data is often more identifiable than it seems.</strong> Even a few
innocuous-looking pieces of information can be enough to identify someone. And
people tend to underestimate what data can be used to reidentify people in a
dataset.</li>
<li><strong>Auxiliary data is a dangerous unknown variable.</strong> Information that seems
secret might be public for certain individuals, or become known to attackers
thanks to an unrelated data breach.</li>
<li><strong>Even "obviously safe" data releases are at risk.</strong> Successful attacks happen
even on datasets that seem well-protected, like aggregated statistics.</li>
<li><strong>Attacks improve over time, in unpredictable ways.</strong> Mitigating only known
attacks, or performing empirical privacy checks, is not enough: using e.g.
newer AI techniques or more powerful hardware can break legacy protections.
<br><br></li>
</ul>
<h1 id="what-to-do-then">What to do, then?</h1>
<p>These failures of legacy techniques prove that we need something better. So,
when does an anonymization method deserve our trust? It should at least address
the four points in the previous section:</p>
<ul>
<li>it should avoid making assumptions on what is identifiable or secret in the
data;</li>
<li>it should be resistant to auxiliary data — its guarantee should hold no matter
what an attacker might already know;</li>
<li>it should provide a mathematical guarantee that doesn't rely on subjective
intuition;</li>
<li>and it should protect against possible future attacks, not just ones known
today.</li>
</ul>
<p>It turns out that this is exactly what <a href="friendly-intro-to-differential-privacy.html">differential privacy</a> provides.</p>
<ul>
<li>It makes no assumptions on what a potential attacker might use in the data.</li>
<li>Its guarantees do not depend on what auxiliary data the attacker has access to.</li>
<li>It provides a quantifiable, provable guarantee about the worst-case privacy risk.</li>
<li>And this guarantee holds for all possible attacks, so the guarantee is future-proof.</li>
</ul>
<p>It has a host of other benefits, too. For example, it can quantify the total
privacy cost of <em>multiple</em> data releases. It also offers much more flexibility:
many kinds of data transformation and analyses can be performed with
differential privacy.</p>Is differential privacy the right fit for your problem?2022-07-18T00:00:00+02:002022-07-18T00:00:00+02:00Damien Desfontainestag:desfontain.es,2022-07-18:/privacy/litmus-test-differential-privacy.html<p>Some data publication or sharing use cases are well-suited to the use of differential privacy, while some aren’t. In this blog post, we give a litmus test allowing you to quickly distinguish between the two.</p><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p> This article was first published on the <a href="https://www.tmlt.io/research/is-differential-privacy-the-right-fit-for-your-problem">Tumult Labs blog</a>; its
copyright is owned by Tumult Labs.</p>
<p></small></p>
<hr>
<p><span class='lettrine'>S</span>ay you have some sensitive data, like a batch of
financial information about first-time home buyers applying for mortgages in
Chicago. You would like to publish this data, or share it with third parties,
for example to facilitate economic research. This financial data has sensitive
information about individuals, so you need to make sure that you’re not
revealing personal data.</p>
<p>So far, this seems like a perfect use case for <a href="friendly-intro-to-differential-privacy.html">differential privacy</a>
(DP): publishing trends without revealing information about individuals is
exactly what it was designed for. You know that DP will successfully protect
this individual data – its guarantees apply regardless of the data distribution.
But you might still be wondering: will I succeed in publishing useful data? Will
it be accurate enough for the people who will use it?</p>
<p>In this blog post, I’ll help you get an initial idea of whether differential
privacy can work for you, using a simple litmus test. Spoiler alert: the process
looks like this.</p>
<p><center>
<img alt="A flowchart representing the litmus test described in the article. It starts
with a question: "How will the data be used?". There are three options.
"I don't know, it's hard to say" leads to "It's complicated. Let's learn more
about the use case first!". "For robust analyses that don't depend too much on
individual data points" leads to "Differential privacy will probably work for
you!". And "For analyses that are very sensitive to tiny differences in the
input data" leads to a second question: "Can you make these analyses more
robust?". If "Yes", then this goes to the same "DP will probably work" box as
earlier. If "Not really", this leads to "Differential privacy will likely not be
a good fit"." src="https://desfontain.es/privacy/images/litmus-test-differential-privacy.png">
</center></p>
<h1 id="a-simple-litmus-test">A simple litmus test</h1>
<p>Consider the decisions people will make based on the published data, and ask
yourself the following question.</p>
<p><center><strong>Can small changes in the original data lead to completely different
decision outcomes?</strong></center></p>
<p>Take the financial data scenario involving first-time home buyer data. Suppose
that a single home-buyer's info was removed from the dataset – would that change
the analysis you are doing on the data? If the analysis is about median value of
mortgages in Chicago overall, probably not. But if the analysis is about the
maximum value of mortgages in just one ZIP code, then removing that maximum
value might change the result by quite a lot!</p>
<p>There are three possible answers to this question.</p>
<ul>
<li>The results of the analysis <strong>do not depend too much on small changes in the
data</strong>. In this case, we say that the data analysis is <strong>robust</strong>, and
<strong>differential privacy will likely work for you</strong>.</li>
<li>The analysis might be <strong>very sensitive to small changes</strong>. In this case, it’s
worth asking: can we make the analysis more robust? If not, then
<strong>differential privacy is likely not a good fit</strong>.</li>
<li>Finally, it might not be clear what the data will be used for, and whether
these analyses will be robust. Then, we need to answer this question first,
and learn more about the use case.</li>
</ul>
<p>Let’s look more closely at these three options.</p>
<h1 id="robust-analyses-well-suited-to-differential-privacy">Robust analyses: well-suited to differential privacy</h1>
<p><em>Robust</em> analyses are those that do not depend too much on individual changes in
the data. Many common data analyses are robust; in particular, almost all
applications that aim at capturing trends fall in that category. For example, if
you are…</p>
<ul>
<li>… estimating large population sizes (> 100)</li>
<li>… understanding correlations between features in a large dataset</li>
<li>… producing usage metrics for a service with many users</li>
<li>… computing statistics over large groups</li>
</ul>
<p>… then the result of these analyses won’t be impacted by very small changes in
the data.</p>
<p><strong>In that case, differential privacy will likely work for you</strong>. Robust analyses
are a particularly good fit for DP techniques: you will likely be able to
generate high-quality data with strong privacy protections. The decisions made
using the DP data will closely resemble those that would have been made on the
true data.</p>
<p>This makes sense: DP is all about adding small amounts of jitter to computations
to hide the data of single individuals. But DP doesn’t need a lot of jitter: the
perturbation’s magnitude is similar to the impact of a single person. If a
single person is unlikely to change the result of future data analyses…
differential privacy probably won’t change it too much, either.</p>
<p>Note that this litmus test tells you about feasibility. It doesn’t always mean
that deploying DP will be very easy. Some use cases, like machine learning, or
situations where you want to release a lot of statistics, can be tricky. In any
case, my colleagues & I at <a href="https://tmlt.io">Tumult Labs</a> can help! Don’t
hesitate to <a href="https://tmlt.io/connect">reach out</a>.</p>
<h1 id="analyses-that-are-sensitive-to-small-changes-in-the-data">Analyses that are sensitive to small changes in the data</h1>
<p>Some analyses are very sensitive to the data of single individuals: a change in
a single person’s data can change the outcome drastically! This typically
happens in three cases.</p>
<h4 id="small-populations">Small populations</h4>
<p>Suppose that you are trying to publish the average mortgage value for a specific
ZIP code, there are only a few first-time home buyers – say, fewer than 10. In
this case, an individual change might have a large impact on the average!</p>
<p>In situations such as this one, individual changes can have a large impact on
the decisions made with the data. In this case, the noise added by differential
privacy is also likely to change the result of the analysis. This will often be
unacceptable: DP will not be a good fit.</p>
<h4 id="finding-outlier-individuals">Finding outlier individuals</h4>
<p>Suppose that you are trying to find which people had mortgages that were
significantly above the average in their area. In applications like this one,
the goal is to detect outlier behavior. This is at odds with the fundamental
goal of differential privacy: hiding information about all individuals,
including outliers! In this kind of scenario, another approach might be
needed.</p>
<h4 id="preserving-linkability">Preserving linkability</h4>
<p>Suppose that you want to enable other people to run analyses joining your data
with their own data, at the level of each individual. In that case, you need a
one-to-one relationship between people in the original data and in the output
data. This is also at odds with differential privacy: you cannot hide
who is present in the sensitive dataset and also preserve linkability. Small
changes in the data will be clearly visible, since one user will or will not be
part of the output.</p>
<h4 id="making-the-analysis-more-robust">Making the analysis more robust</h4>
<p>When the analysis is sensitive to small changes in the data, it is worth asking:
could we change that? Can we reformulate the problem in a more robust way? Doing
so can often be doubly beneficial, and lead to privacy <em>and</em> utility
improvements.</p>
<p>Say that the published data will be used to determine the impact of age on the
rejection rate for mortgages. A first approach would be to release rejection
rates, grouped by age. But some values of age are rarer than others: we might
have many data points where the age is 40, but only a handful where the age is
20.</p>
<p>For these outlier values, small changes in the data might lead to large changes.
But we are not interested in specific age values, only about the global
relationship between age and rejection rate. Thus, we could change our strategy
to publish data for age ranges, so each statistic comes from more data, and is
more robust to small changes. This would make the released data more
trustworthy, and the publication process more amenable to differential privacy.</p>
<p>For use cases that appear to require linkability, ask yourself the question: can
we perform the join between datasets before computing the statistics of
interest? If so, then using differential privacy might be an option.</p>
<h1 id="what-if-i-dont-know-how-the-data-will-be-used">What if I don’t know how the data will be used?</h1>
<p>Sometimes, the question from our litmus test might be difficult to answer: what
decisions will be made based on the published data? You might know that other
people want access to this data, but not know exactly what they will want to do
with it.</p>
<p>The right thing to do, then, is to try and understand more about their use case.
The more you know what they want to do, the easier it will be to design a
solution that works for them. This is both to answer our question about
feasibility, and to help craft the requirements for a possible DP-based
solution. The more you understand the requirements of your stakeholders, the
happier you and they will be with the released data.</p>
<hr>
<p><small></p>
<p>Thanks to Ashwin Machanavajjhala, Gerome Miklau, and Nicole Le for helpful
feedback on this post.</p>
<p></small></p>Research post: Differential privacy under partial knowledge2022-06-27T00:00:00+02:002022-06-27T00:00:00+02:00Damien Desfontainestag:desfontain.es,2022-06-27:/privacy/partial-knowledge.html<p>What happens to differential privacy if you model a weaker adversary with only partial knowledge over the input data?</p><p><strong>Differential</strong> privacy (DP) makes very few assumptions on the attacker. The
definition holds for all pairs of databases <span class="math">\(D_1\)</span> and <span class="math">\(D_2\)</span> that differ in one
person. This means that even if the attacker knows <em>everyone</em> in the database,
except one person, they can't get information about this person. Or, rather, the
probabilistic information they can get is bounded by <span class="math">\(e^\varepsilon\)</span>.</p>
<p>When explaining DP to people for the first time, this "perfect knowledge"
assumption often raises eyebrows. This seems overkill, right? If someone knows
<em>everyone</em> in your database except one person… This probably means that they
have direct access to your raw data. And in that case, you've already lost! It
no longer matters how you're anonymizing the data later.</p>
<p>This intuition was central to my PhD proposal, back in early 2016. Then, I was
observing two different worlds. In academia, researchers focused their efforts
on differential privacy. Definitions like <a href="k-anonymity.html"><span class="math">\(k\)</span>-anonymity</a> were
a thing of the past. In industry, however, DP was still largely ignored. People
thought it was a neat theoretical idea, but pretty much unusable in practice.
Its strong assumptions were unrealistic, and the idea of adding noise to data
was too painful.</p>
<p>So, I thought: I'm trying to start a <a href="part-time-phd.html">part-time PhD</a>,
splitting my weeks between industry and academia. Can I work towards bridging
this gap? What if we considered DP, but made its assumptions a little weaker…
Would it be possible to prove something about older notions like <span class="math">\(k\)</span>-anonymity?
They might not be as robust as DP, but they might be good enough sometimes,
right? Could one, for example, quantify their guarantees using the same
<a href="differential-privacy-in-more-detail.html">Bayesian interpretation</a> as with DP?</p>
<p>This line of study eventually led to <a href="https://arxiv.org/abs/1905.00650">this paper</a>, a collaboration with
<a href="https://mohammadi.eu/">Esfandiar Mohammadi</a>, Elisabeth Krahmer, and my PhD advisor <a href="https://people.inf.ethz.ch/basin/">David
Basin</a>. We studied this natural question: what if an attacker only has
<em>partial</em> knowledge over the data? How can we formalize this assumption? How
does the DP definition change, and can we use this variant to prove interesting
results?</p>
<p>This basic idea is not new: the formalism we used was based on work by <a href="https://eprint.iacr.org/2011/487.pdf">Bhaskar
et al.</a>, published in 2011. To capture the attacker's uncertainty, we
model the input data by a <em>probability distribution</em>, denoted <span class="math">\(\theta\)</span>. And
instead of comparing the output of the mechanism <span class="math">\(A\)</span> on two fixed databases, we
compare it on two samples from this distribution.</p>
<p><center>
<svg version="1.1"
viewBox="0.0 0.0 1154.6010498687665 478.09711286089237"
aria-label="Diagram illustrating the definition of DP with partial knowledge. A cloud on the left is labeled "Data-generating distribution θ". Two arrows start from it: going to two databases of different colors; the arrows have separate labels "Conditioned on record i being X", with X being two rows of distinct colors. The databases both have an arrow going to a silly "magic" gif, and this has an arrow going to a cloud labeled "Data". A double arrow labeled "basically the same" points to the two outputs."
fill="none"
stroke="none"
stroke-linecap="square"
stroke-miterlimit="10"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns="http://www.w3.org/2000/svg">
<clipPath id="p.0">
<path d="m0 0l1154.6011 0l0 478.0971l-1154.6011 0l0 -478.0971z"
clip-rule="nonzero" />
</clipPath>
<g clip-path="url(#p.0)">
<path fill="#000000" fill-opacity="0.0"
d="m0 0l1154.6011 0l0 478.0971l-1154.6011 0z"
fill-rule="evenodd" />
<path fill="#f6cd4c"
d="m896.73236 65.35537l0 0c-1.8811646 -14.366959 4.295044 -28.589382 15.907898 -36.632164c11.612793 -8.042784 26.62555 -8.495398 38.667786 -1.1657829l0 0c4.265686 -8.353548 12.073242 -14.121096 21.060913 -15.558073c8.987732 -1.4369774 18.099976 1.6253939 24.580444 8.260807l0 0c3.63385 -7.573864 10.768982 -12.662533 18.873474 -13.460281c8.104492 -0.7977495 16.031372 2.808321 20.967773 9.538591l0 0c6.5650635 -8.028276 17.010254 -11.408514 26.815918 -8.678066c9.805664 2.7304478 17.210571 11.081169 19.010498 21.438759l0 0c8.043335 2.2800636 14.743286 8.076288 18.368774 15.891115c3.6254883 7.814831 3.8209229 16.881641 0.53564453 24.857868l0 0c7.92041 10.712936 9.773071 24.9878 4.8669434 37.49746c-4.90625 12.509666 -15.834351 21.374687 -28.706177 23.286812c-0.09082031 11.740753 -6.286621 22.513947 -16.19934 28.167084c-9.912842 5.6531525 -21.994629 5.3035126 -31.588745 -0.9141693l0 0c-4.086426 14.0616 -15.588745 24.40799 -29.537292 26.569153c-13.948547 2.161148 -27.842834 -4.250351 -35.680054 -16.464508l0 0c-9.606812 6.020401 -21.134155 7.7546844 -31.98169 4.8116455c-10.847595 -2.9430542 -20.101624 -10.315521 -25.674622 -20.45433l0 0c-9.816895 1.1938782 -19.308533 -4.091919 -23.76416 -13.23407c-4.4556274 -9.142136 -2.9268188 -20.194473 3.8277588 -27.671783l0 0c-8.756958 -5.3563843 -13.225281 -15.9852295 -11.074951 -26.34401c2.1503906 -10.35878 10.432129 -18.100159 20.526672 -19.18731z"
fill-rule="evenodd" />
<path fill="#000000" fill-opacity="0.0"
d="m887.08655 111.440414l0 0c4.1324463 2.5276947 8.906372 3.6743164 13.680786 3.2859116m6.25354 37.620857c2.0531616 -0.24969482 4.065674 -0.77845764 5.9856567 -1.5726471m51.667725 17.214676c-1.4440918 -2.2505646 -2.6531372 -4.6555023 -3.6065674 -7.173813m68.82648 -2.931244l0 0c0.7449951 -2.5636444 1.2277832 -5.202179 1.4400635 -7.8715973m46.34607 -19.38034c0.09667969 -12.499863 -6.7348633 -23.944855 -17.56018 -29.41903m41.399292 -31.36393c-1.7530518 4.256439 -4.4294434 8.032265 -7.819214 11.031433m-11.083496 -51.78153l0 0c0.29870605 1.7188435 0.43688965 3.4635239 0.41271973 5.209854m-46.238403 -17.970028l0 0c-1.6376953 2.002655 -2.9869385 4.240597 -4.0057373 6.644142m-35.835876 -2.7233276l0 0c-0.8727417 1.8189983 -1.5244751 3.7438087 -1.9401245 5.730093m-43.701965 1.5673256l0 0c2.5481567 1.5509415 4.9055176 3.4176655 7.0203247 5.559164m-61.595276 32.238983l0 0c0.25927734 1.9801407 0.6689453 3.9358673 1.2251587 5.848877"
fill-rule="evenodd" />
<path fill="#7c7ce0"
d="m736.979 81.62992l114.703186 0l0 -13.535431l16.5094 27.070862l-16.5094 27.07087l0 -13.535431l-114.703186 0z"
fill-rule="evenodd" />
<path fill="#000000" fill-opacity="0.0"
d="m932.9249 68.09998l118.8031 0l0 54.128647l-118.8031 0z"
fill-rule="evenodd" />
<g transform="matrix(0.15289973753280842 0.0 0.0 0.15290577427821522 932.9249446194225 68.09998320209974)">
<clipPath id="p.1">
<path d="m0 -5.684342E-14l777.0 0l0 354.0l-777.0 0z"
clip-rule="evenodd" />
</clipPath>
<image clip-path="url(#p.1)" fill="#000" width="777.0"
height="354.0" x="0.0" y="0.0" preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/data.png"/>
</g>
<path fill="#f6cd4c"
d="m896.73236 350.73334l0 0c-1.8811646 -14.366974 4.295044 -28.589386 15.907898 -36.63217c11.612793 -8.042786 26.62555 -8.495392 38.667786 -1.165802l0 0c4.265686 -8.353546 12.073242 -14.121094 21.060913 -15.558044c8.987732 -1.4369812 18.099976 1.6253662 24.580444 8.260803l0 0c3.63385 -7.573883 10.768982 -12.662537 18.873474 -13.460297c8.104492 -0.79776 16.031372 2.808319 20.967773 9.538605l0 0c6.5650635 -8.02829 17.010254 -11.408508 26.815918 -8.67807c9.805664 2.7304382 17.210571 11.081177 19.010498 21.438751l0 0c8.043335 2.2800598 14.743286 8.076294 18.368774 15.891113c3.6254883 7.8148193 3.8209229 16.881653 0.53564453 24.85788l0 0c7.92041 10.712921 9.773071 24.987793 4.8669434 37.497467c-4.90625 12.509644 -15.834351 21.374664 -28.706177 23.286804c-0.09082031 11.740753 -6.286621 22.513947 -16.19934 28.167084c-9.912842 5.653137 -21.994629 5.3034973 -31.588745 -0.9141846l0 0c-4.086426 14.061615 -15.588745 24.40799 -29.537292 26.569153c-13.948547 2.1611633 -27.842834 -4.2503357 -35.680054 -16.464508l0 0c-9.606812 6.0204163 -21.134155 7.7546997 -31.98169 4.8116455c-10.847595 -2.9430542 -20.101624 -10.315521 -25.674622 -20.454315l0 0c-9.816895 1.1938782 -19.308533 -4.091919 -23.76416 -13.23407c-4.4556274 -9.142151 -2.9268188 -20.194489 3.8277588 -27.671783l0 0c-8.756958 -5.3563843 -13.225281 -15.9852295 -11.074951 -26.344025c2.1503906 -10.358765 10.432129 -18.100159 20.526672 -19.187317z"
fill-rule="evenodd" />
<path fill="#000000" fill-opacity="0.0"
d="m887.08655 396.81836l0 0c4.1324463 2.52771 8.906372 3.6743164 13.680786 3.2859192m6.25354 37.62085c2.0531616 -0.24969482 4.065674 -0.7784424 5.9856567 -1.5726318m51.667725 17.21466c-1.4440918 -2.2505493 -2.6531372 -4.655487 -3.6065674 -7.1737976m68.82648 -2.931244l0 0c0.7449951 -2.5636597 1.2277832 -5.202179 1.4400635 -7.8716125m46.34607 -19.38034c0.09667969 -12.499847 -6.7348633 -23.944855 -17.56018 -29.419037m41.399292 -31.363922c-1.7530518 4.256439 -4.4294434 8.032257 -7.819214 11.031433m-11.083496 -51.781525l0 0c0.29870605 1.7188416 0.43688965 3.4635315 0.41271973 5.2098694m-46.238403 -17.970032l0 0c-1.6376953 2.002655 -2.9869385 4.2406006 -4.0057373 6.6441345m-35.835876 -2.7233276l0 0c-0.8727417 1.8190002 -1.5244751 3.743805 -1.9401245 5.7301025m-43.701965 1.5673218l0 0c2.5481567 1.5509338 4.9055176 3.4176636 7.0203247 5.5591736m-61.595276 32.238983l0 0c0.25927734 1.980133 0.6689453 3.935852 1.2251587 5.848877"
fill-rule="evenodd" />
<path fill="#000000" fill-opacity="0.0"
d="m932.9249 353.47794l118.8031 0l0 54.12863l-118.8031 0z"
fill-rule="evenodd" />
<g transform="matrix(0.15289973753280842 0.0 0.0 0.15290577427821522 932.9249446194225 353.4779359580052)">
<clipPath id="p.2">
<path d="m0 0l777.0 0l0 354.0l-777.0 0z"
clip-rule="evenodd" />
</clipPath>
<image clip-path="url(#p.2)" fill="#000" width="777.0"
height="354.0" x="0.0" y="0.0" preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/data.png"/>
</g>
<path fill="#db4437"
d="m959.42914 190.68767l32.897644 -32.897644l32.897583 32.897644l-16.448792 0l0 94.3307l16.448792 0l-32.897583 32.897644l-32.897644 -32.897644l16.448792 0l0 -94.3307z"
fill-rule="evenodd" />
<path fill="#db4437"
d="m837.7743 209.5912c51.51703 -13.457993 103.03412 13.458008 154.55115 0c51.51709 -13.457993 103.03418 13.458008 154.55115 0l0 56.523636c-51.516968 13.458008 -103.03406 -13.458008 -154.55115 0c-51.51703 13.458008 -103.03412 -13.458008 -154.55115 0z"
fill-rule="evenodd" />
<path fill="#000000"
d="m890.6053 252.63303q-1.484375 -0.96875 -1.75 -1.953125q-0.25 -0.96875 0.671875 -1.546875q0.21875 -0.140625 0.8125 -0.03125q0.609375 0.09375 0.921875 0.40625q0.703125 0.625 1.578125 0.78125q0.890625 0.15625 2.171875 -0.1875q1.453125 -0.453125 2.234375 -1.125q0.78125 -0.6875 1.75 -1.859375q0.734375 -0.9375 1.015625 -1.828125q0.296875 -0.90625 0.28125 -1.65625q-0.015625 -0.75 -0.359375 -1.125q-0.875 -1.21875 -2.15625 -1.671875q-1.28125 -0.46875 -3.359375 0.078125q-0.234375 0.03125 -0.71875 0.140625q-0.484375 0.109375 -0.875 0.203125q-0.375 0.078125 -0.40625 0.078125q-0.25 0.0625 -0.390625 0.0625q-0.140625 0 -0.328125 0.328125q-0.1875 0.328125 -0.578125 1.359375q-0.375 1.015625 -1.140625 3.125q-0.71875 2.125 -1.234375 3.015625q-0.5 0.875 -0.875 0.796875q-0.375 -0.09375 -0.859375 -0.75q-0.28125 -0.5 -0.609375 -1.078125q-0.328125 -0.59375 0.015625 -1.21875q0.171875 -0.3125 0.265625 -0.5625q0.109375 -0.265625 0.28125 -0.921875q0.171875 -0.515625 0.671875 -1.8125q0.515625 -1.3125 1.203125 -3.0625q0.6875 -1.75 1.453125 -3.609375q0.765625 -1.875 1.46875 -3.53125q0.71875 -1.671875 1.203125 -2.828125q0.484375 -1.0625 0.578125 -1.546875q0.109375 -0.484375 -0.0625 -0.53125q0 -0.171875 -0.109375 -0.328125q-0.09375 -0.15625 -0.0625 -0.296875q0 -0.125 0.609375 -0.359375q0.609375 -0.25 1.296875 -0.453125q0.703125 -0.21875 1.015625 -0.15625q0.171875 0 0.40625 -0.015625q0.25 -0.015625 0.25 -0.15625q0.09375 0 0.640625 -0.0625q0.5625 -0.078125 1.21875 0q2.328125 0.203125 3.8125 1.328125q1.5 1.125 1.46875 3.59375q-0.046875 0.59375 -0.5 1.53125q-0.4375 0.921875 -1.125 1.859375q-0.671875 0.9375 -1.4375 1.625q-0.4375 0.390625 -0.953125 0.796875q-0.5 0.390625 -0.859375 0.65625q-0.359375 0.25 -0.359375 0.25q0 0 0.265625 0.109375q0.28125 0.09375 0.59375 0.203125q1.59375 0.59375 2.515625 1.5q0.921875 0.890625 1.296875 2.484375q0.140625 0.515625 0.015625 1.28125q-0.125 0.765625 -0.328125 1.515625q-0.203125 0.734375 -0.453125 1.265625q-0.234375 0.234375 -0.375 0.5q-0.140625 0.25 -0.140625 0.25q0 0.078125 -0.28125 0.546875q-0.265625 0.46875 -0.65625 1.03125q-0.375 0.546875 -0.71875 0.953125q-0.359375 0.28125 -1.0 0.78125q-0.640625 0.5 -1.390625 0.984375q-0.734375 0.5 -1.390625 0.734375q-0.96875 0.421875 -2.171875 0.578125q-1.1875 0.15625 -2.265625 0.03125q-1.078125 -0.125 -1.703125 -0.5zm2.5 -14.390625q0 0 0.5625 -0.109375q0.578125 -0.125 1.265625 -0.28125q0.703125 -0.15625 1.046875 -0.296875q0.171875 -0.109375 0.5 -0.28125q0.328125 -0.171875 0.671875 -0.515625q0.5 -0.453125 1.046875 -1.015625q0.546875 -0.578125 1.078125 -1.234375q0.546875 -0.65625 0.9375 -1.421875q0.796875 -1.25 0.859375 -1.796875q0.078125 -0.5625 -0.375 -1.015625q-0.3125 -0.375 -0.875 -0.5625q-0.546875 -0.203125 -1.1875 -0.21875q-0.640625 -0.015625 -1.234375 0.15625q-0.453125 0.140625 -0.703125 0.328125q-0.234375 0.1875 -0.234375 0.578125q0 0.3125 -0.1875 0.640625q-0.1875 0.328125 -0.359375 0.5q-0.046875 0.25 -0.234375 0.46875q-0.1875 0.21875 -0.1875 0.21875q0 0.140625 -0.4375 0.859375q-0.421875 0.703125 -0.921875 2.015625q-0.609375 1.5 -0.828125 2.15625q-0.203125 0.65625 -0.203125 0.828125zm12.890869 11.15625q-0.140625 -0.265625 -0.609375 -0.546875q-0.46875 -0.28125 -0.46875 -0.421875q0 -0.140625 -0.15625 -0.421875q-0.15625 -0.296875 -0.296875 -0.4375q-0.171875 -0.203125 0.109375 -1.015625q0.28125 -0.828125 0.828125 -2.0q0.5625 -1.1875 1.21875 -2.390625q0.28125 -0.28125 0.546875 -0.734375q0.28125 -0.453125 0.421875 -0.578125q0 -0.140625 0.09375 -0.265625q0.109375 -0.125 0.25 -0.125l0.28125 -0.546875q0.0625 -0.140625 0.421875 -0.546875q0.375 -0.421875 0.890625 -0.921875q0.53125 -0.515625 1.0625 -0.953125q0.53125 -0.453125 0.875 -0.671875q0.734375 -0.484375 1.515625 -0.4375q0.78125 0.03125 1.328125 0.484375q0.109375 0.09375 0.46875 0.40625q0.375 0.3125 0.828125 0.703125q0.46875 0.375 0.828125 0.6875l1.28125 1.140625l-0.59375 1.734375q-0.453125 1.390625 -0.234375 2.4375q0.234375 1.03125 0.65625 1.796875q0.171875 0.375 0.453125 0.59375q0.296875 0.203125 0.78125 0.265625q0.28125 0.03125 0.515625 0.203125q0.25 0.171875 0.25 0.5625q0 0.828125 -0.296875 1.203125q-0.28125 0.359375 -0.640625 0.421875q-1.140625 0.140625 -2.1875 -0.234375q-1.03125 -0.390625 -1.859375 -1.984375q-0.140625 -0.203125 -0.375 -0.765625q-0.21875 -0.578125 -0.25 -0.765625q-0.140625 0.1875 -0.375 0.453125q-0.21875 0.25 -0.421875 0.453125q-2.46875 2.5 -4.21875 3.25q-1.75 0.734375 -2.921875 -0.03125zm1.90625 -2.875q0.25 0.046875 0.765625 -0.25q0.515625 -0.296875 1.390625 -1.0q1.171875 -1.0 1.9375 -1.65625q0.765625 -0.65625 1.484375 -1.703125l0.765625 -1.25q-0.34375 -0.796875 -0.625 -1.03125q-0.28125 -0.25 -0.546875 -0.25q-0.453125 0 -1.078125 0.578125q-0.625 0.578125 -1.40625 1.671875q-0.78125 1.078125 -1.75 2.71875q-0.453125 0.75 -0.703125 1.46875q-0.234375 0.703125 -0.234375 0.703125zm14.550476 3.8125q-1.171875 -0.484375 -1.71875 -1.015625q-0.53125 -0.546875 -0.421875 -0.953125q0.0625 -0.203125 0.234375 -0.375q0.171875 -0.1875 0.78125 -0.15625q0.609375 0.015625 1.890625 0.328125q0.390625 0 0.765625 -0.03125q0.375 -0.046875 0.703125 -0.109375q0.328125 -0.078125 0.546875 -0.171875q0.171875 -0.078125 0.09375 -0.1875q-0.0625 -0.125 -0.234375 -0.265625q-0.140625 -0.171875 -0.3125 -0.21875q-0.171875 -0.0625 -0.3125 -0.0625q-0.0625 -0.171875 -0.296875 -0.34375q-0.21875 -0.171875 -0.53125 -0.390625q-0.765625 -0.609375 -1.390625 -1.09375q-0.625 -0.5 -0.484375 -0.734375q0 -0.140625 0 -0.15625q0 -0.015625 0 -0.015625q-0.25 0 -0.453125 -0.671875q-0.203125 -0.6875 -0.203125 -1.234375q0 -0.703125 0.296875 -1.390625q0.296875 -0.6875 0.75 -1.3125q0.46875 -0.625 0.953125 -1.109375q0.671875 -0.625 1.40625 -1.234375q0.75 -0.609375 1.625 -1.09375q0.890625 -0.484375 1.96875 -0.765625q0.484375 -0.09375 1.109375 -0.09375q0.625 0 0.90625 0.171875q0.265625 -0.171875 0.6875 0.09375q0.421875 0.25 0.828125 0.703125q0.421875 0.453125 0.625 0.90625q0 0.40625 -0.015625 1.015625q-0.015625 0.609375 -0.09375 1.0625q-0.0625 0.4375 -0.3125 0.4375q0 0 -0.09375 0.03125q-0.078125 0.015625 -0.078125 0.1875q0 0.34375 -0.515625 0.796875q-0.65625 0.453125 -1.25 0.734375q-0.578125 0.265625 -0.890625 0.03125q-0.21875 -0.140625 -0.390625 -0.28125q-0.171875 -0.140625 -0.3125 -0.34375q0 0 -0.09375 -0.15625q-0.078125 -0.15625 -0.078125 -0.296875q0 -0.140625 0.09375 -0.265625q0.109375 -0.125 0.109375 -0.125q0.484375 -0.4375 0.75 -0.8125q0.265625 -0.390625 0.359375 -0.6875q0.109375 -0.296875 0.109375 -0.46875q0 -0.203125 -0.09375 -0.25q-0.078125 -0.0625 -0.359375 -0.0625q-0.171875 0 -0.75 0.28125q-0.5625 0.28125 -1.1875 0.6875q-0.625 0.390625 -0.96875 0.703125q-0.140625 0.234375 -0.203125 0.3125q-0.046875 0.0625 -0.171875 0.0625q0 -0.140625 -0.1875 0.03125q-0.171875 0.171875 -0.421875 0.484375q-0.234375 0.3125 -0.484375 0.65625q-0.234375 0.34375 -0.375 0.5625q-0.03125 0.3125 -0.15625 0.578125q-0.109375 0.25 -0.078125 0.53125q0.03125 0.34375 0.34375 0.65625q0.3125 0.3125 0.90625 0.796875q1.203125 0.90625 1.8125 1.421875q0.609375 0.515625 0.84375 0.828125q0.25 0.3125 0.3125 0.625q0.59375 1.078125 0.140625 1.859375q-0.453125 0.78125 -1.375 1.1875q-0.359375 0.171875 -1.234375 0.328125q-0.859375 0.15625 -1.8125 0.140625q-0.953125 -0.015625 -1.609375 -0.296875zm12.7547 -0.234375q-0.828125 0 -1.4375 -0.875q-0.609375 -0.890625 -0.734375 -2.140625q0.03125 -0.28125 0.078125 -0.796875q0.046875 -0.515625 0.125 -1.078125q0.140625 -0.65625 0.375 -1.6875q0.25 -1.046875 0.515625 -2.09375q0.28125 -1.0625 0.484375 -1.84375q0.21875 -0.78125 0.25 -0.921875q0.203125 -0.484375 0.9375 -0.234375q0.734375 0.234375 1.25 1.03125q0.203125 0.28125 0.25 0.65625q0.0625 0.375 -0.09375 1.109375q-0.15625 0.734375 -0.578125 2.140625q-0.375 1.390625 -0.578125 2.25q-0.1875 0.84375 -0.140625 1.78125q-0.140625 1.59375 -0.28125 2.171875q-0.140625 0.5625 -0.421875 0.53125zm3.1875 -14.390625q-0.140625 0.109375 -0.265625 0.09375q-0.109375 -0.015625 -0.390625 -0.1875q-0.234375 -0.046875 -0.609375 -0.328125q-0.359375 -0.296875 -0.625 -0.609375q-0.25 -0.3125 -0.25 -0.390625q-0.03125 -0.34375 0.046875 -1.09375q0.09375 -0.765625 0.375 -1.4375q0.296875 -0.6875 0.828125 -0.71875q0.140625 0 0.59375 0.140625q0.46875 0.140625 1.0 0.34375q0.265625 0.515625 0.5 1.359375q0.25 0.828125 0.0625 1.609375q-0.1875 0.765625 -1.265625 1.21875zm4.1988525 14.140625q-0.765625 0 -1.125 -0.0625q-0.359375 -0.078125 -0.546875 -0.203125q-0.171875 -0.140625 -0.4375 -0.390625q-0.140625 -0.28125 -0.5 -0.703125q-0.34375 -0.4375 -0.46875 -0.578125q-0.140625 -0.515625 -0.09375 -1.421875q0.046875 -0.90625 0.28125 -1.9375q0.25 -1.046875 0.609375 -1.875q0.125 -0.28125 0.234375 -0.546875q0.109375 -0.28125 0.109375 -0.28125q0 -0.171875 0.421875 -0.828125q0.4375 -0.65625 0.96875 -1.390625q0.546875 -0.734375 0.890625 -1.046875q0.25 -0.265625 0.75 -0.578125q0.5 -0.3125 0.875 -0.453125q0.390625 -0.140625 1.046875 -0.359375q0.65625 -0.234375 1.171875 -0.234375q0.484375 0 1.03125 0.21875q0.5625 0.203125 0.953125 0.71875q0.625 0.625 0.796875 1.5q0.171875 0.859375 0.03125 1.484375q-0.171875 0.6875 -0.703125 1.125q-0.515625 0.4375 -1.109375 0.453125q-0.578125 0.015625 -1.03125 -0.46875q-0.140625 -0.28125 -0.140625 -0.46875q0 -0.1875 0.140625 -0.578125q0.140625 -0.484375 0.140625 -0.859375q0 -0.390625 -0.3125 -0.390625q-0.453125 0 -0.734375 0.265625q-0.265625 0.265625 -0.828125 0.84375q-0.234375 0.28125 -0.625 0.8125q-0.375 0.515625 -0.75 1.125q-0.359375 0.59375 -0.625 1.21875q-0.390625 1.140625 -0.484375 1.484375q-0.078125 0.328125 -0.078125 0.328125q0 0.125 -0.03125 0.25q-0.03125 0.125 -0.03125 0.265625q0 0.40625 0.484375 0.65625q0.375 0.234375 0.625 0.21875q0.265625 -0.015625 1.03125 -0.359375q0.5625 -0.28125 1.0625 -0.71875q0.5 -0.453125 1.046875 -0.8125q0.390625 -0.234375 0.609375 -0.40625q0.234375 -0.171875 0.234375 -0.03125q0 0.140625 0.09375 0.140625q0.109375 0 0.25 -0.140625q0.203125 -0.28125 0.515625 -0.21875q0.3125 0.046875 0.5625 0.1875q0.265625 0.125 0.265625 0.234375q-0.0625 0.3125 0.140625 0.515625q0 0.140625 -0.09375 0.375q-0.078125 0.21875 -0.21875 0.359375q-0.171875 0.234375 -0.5 0.6875q-0.328125 0.453125 -0.609375 0.796875q-0.265625 0.34375 -0.265625 0.34375q-0.140625 -0.03125 -0.265625 0.078125q-0.125 0.09375 -0.265625 0.234375q-0.265625 0.25 -0.515625 0.375q-0.234375 0.109375 -0.375 0.109375q-0.25 0.140625 -0.375 0.1875q-0.109375 0.03125 -0.109375 0.171875q-0.359375 0.34375 -1.265625 0.46875q-0.890625 0.109375 -1.859375 0.109375zm11.476257 -0.34375q-0.421875 0.109375 -0.9375 -0.234375q-0.515625 -0.359375 -0.9375 -0.875q-0.421875 -0.515625 -0.515625 -0.9375q-0.140625 -0.71875 0.078125 -1.78125q0.234375 -1.0625 0.78125 -2.25q0.5625 -1.203125 1.390625 -2.359375q0.828125 -1.15625 1.859375 -2.09375q1.046875 -0.9375 2.265625 -1.453125q0.515625 -0.28125 0.71875 -0.359375q0.21875 -0.09375 0.421875 0q0.203125 0.078125 0.734375 0.359375q0.484375 0.140625 0.984375 0.328125q0.5 0.1875 0.875 0.703125q0.28125 0.515625 0.421875 1.046875q0.140625 0.515625 0.234375 1.171875q0.28125 0.875 0.203125 1.140625q-0.0625 0.25 -0.203125 0.390625q-0.265625 0.34375 -0.125 0.34375q0 0 -0.03125 0.125q-0.015625 0.109375 -0.15625 0.359375q-0.234375 0.765625 -0.359375 1.4375q-0.125 0.671875 0.015625 1.203125q0.28125 0.75 0.75 0.890625q0.46875 0.140625 1.234375 -0.625q0.375 -0.265625 0.859375 -0.359375q0.484375 -0.09375 0.484375 0.1875q0 0 0.109375 0.09375q0.109375 0.078125 0.234375 0.078125q0.171875 0 0.140625 0.328125q-0.03125 0.328125 -0.140625 0.75q-0.09375 0.421875 -0.265625 0.6875q-0.53125 0.53125 -0.53125 0.703125q0 0 -0.125 0.125q-0.109375 0.109375 -0.390625 0.359375q-0.625 0.375 -1.578125 0.375q-0.953125 0 -1.46875 -0.375q-0.515625 -0.390625 -1.03125 -0.9375q-0.5 -0.5625 -0.640625 -1.046875l-0.171875 -0.34375l-1.0 0.65625q-0.375 0.390625 -0.5625 0.578125q-0.171875 0.1875 -0.296875 0.328125q-0.109375 0.140625 -0.234375 0.25q-0.125 0.09375 -0.25 0.09375q-0.140625 0 -0.25 0q-0.109375 0 -0.109375 0q0 0.140625 -0.484375 0.375q-0.484375 0.21875 -1.078125 0.390625q-0.578125 0.171875 -0.921875 0.171875zm0.71875 -2.90625q0.21875 0 0.640625 -0.265625q0.4375 -0.265625 0.953125 -0.640625q0.53125 -0.375 0.953125 -0.75q0.4375 -0.390625 0.640625 -0.640625q0.140625 -0.125 0.328125 -0.4375q0.203125 -0.3125 0.4375 -0.5625q0.796875 -0.90625 1.0 -1.4375q0.21875 -0.53125 -0.171875 -1.15625q-0.140625 -0.390625 -0.34375 -0.421875q-0.203125 -0.03125 -0.859375 0.25q-0.421875 0.234375 -0.90625 0.796875q-0.484375 0.546875 -0.984375 1.25q-0.5 0.6875 -0.921875 1.421875q-0.421875 0.71875 -0.6875 1.3125q-0.25 0.578125 -0.25 0.9375q0 0.125 0.078125 0.234375q0.09375 0.109375 0.09375 0.109375zm11.989746 4.578125q-0.453125 -0.1875 -0.734375 -0.546875q-0.265625 -0.359375 -0.296875 -1.046875q-0.03125 -0.453125 0.03125 -1.328125q0.0625 -0.890625 0.25 -1.796875q0 -0.078125 0.015625 -0.15625q0.015625 -0.09375 0.015625 -0.125q0.03125 -0.140625 0.0625 -0.421875q0.03125 -0.296875 0.078125 -0.484375q0.0625 -0.203125 0.0625 -0.03125l0.203125 -0.9375q0.140625 -0.625 0.3125 -1.28125q0.171875 -0.65625 0.328125 -1.15625q0.15625 -0.5 0.203125 -0.578125q0 0 0.109375 -0.390625q0.125 -0.40625 0.265625 -0.8125q0.546875 -1.734375 1.046875 -2.875q0.515625 -1.15625 0.890625 -1.671875q0.03125 -0.09375 0.09375 -0.25q0.078125 -0.15625 0.125 -0.265625q0 -0.375 0.046875 -0.546875q0.046875 -0.1875 0.15625 -0.3125q0.0625 -0.140625 0.203125 -0.359375q0.140625 -0.234375 0.3125 -0.578125l0.140625 -0.390625q0.203125 -0.453125 0.515625 -1.046875q0.3125 -0.609375 0.453125 -0.921875q0.109375 -0.3125 0.421875 -0.75q0.3125 -0.4375 0.6875 -0.5625l-0.03125 -0.046875q0.171875 -0.3125 0.65625 -0.5q0.484375 -0.1875 0.796875 -0.1875q0.484375 0 0.734375 0.28125q0.265625 0.265625 0.265625 0.75q0 0.53125 -0.015625 0.765625q-0.015625 0.21875 -0.25 0.484375l0.03125 0q0.0625 -0.0625 -0.046875 0.21875q-0.09375 0.265625 -0.171875 0.375q-0.0625 0.171875 -0.15625 0.390625q-0.078125 0.203125 -0.1875 0.296875q-0.09375 0.140625 -0.546875 0.96875q-0.453125 0.8125 -1.015625 1.90625q-0.546875 1.078125 -1.046875 2.109375q-0.5 1.015625 -0.71875 1.578125l-0.140625 0.515625q-0.0625 0.140625 -0.0625 0.203125q-0.109375 0.25 -0.21875 0.5q-0.09375 0.234375 -0.203125 0.40625l-0.546875 2.140625l-0.140625 0.734375l-0.3125 0.796875l-0.203125 0.65625q-0.078125 0.203125 -0.109375 0.453125q-0.03125 0.234375 -0.140625 0.34375q0 0.03125 -0.015625 0.09375q-0.015625 0.046875 -0.015625 0.078125q-0.140625 0.5625 -0.46875 1.5625q-0.328125 1.0 -0.390625 2.21875q0 0.03125 -0.03125 0.078125q-0.015625 0.03125 -0.015625 0.0625q-0.09375 0.71875 -0.421875 1.109375q-0.328125 0.40625 -0.859375 0.28125zm7.3875732 -0.671875q-0.65625 0.171875 -1.1875 -0.359375q-0.515625 -0.53125 -0.71875 -1.5625q-0.203125 -1.046875 0.0625 -2.328125q0.171875 -0.421875 0.28125 -0.8125q0.109375 -0.40625 0.109375 -0.65625q0.0625 -0.34375 0.265625 -1.015625q0.21875 -0.671875 0.453125 -1.390625q0.25 -0.734375 0.46875 -1.3125q0.234375 -0.578125 0.296875 -0.703125q0.140625 0 0.234375 -0.25q0.109375 -0.25 0.109375 -0.484375q0 -0.28125 0.125 -0.625q0.125 -0.34375 0.265625 -0.484375q0 -0.171875 0.078125 -0.390625q0.09375 -0.234375 0.09375 -0.375q0 -0.140625 0.078125 -0.265625q0.09375 -0.140625 0.09375 -0.28125q0.234375 -0.453125 0.625 -1.4375q0.40625 -1.0 0.875 -2.203125q0.46875 -1.21875 0.890625 -2.390625q0.421875 -1.1875 0.671875 -2.0q0.265625 -0.8125 0.265625 -0.984375q0 -0.28125 0.203125 -0.515625q0.203125 -0.25 0.484375 -0.25q0.109375 -0.09375 0.484375 -0.140625q0.390625 -0.0625 0.53125 -0.0625q0 0 0.125 0.140625q0.140625 0.140625 0.421875 0.40625q0.28125 0.25 0.53125 0.5625q0.265625 0.3125 0.265625 0.71875q0 0.453125 -0.125 0.875q-0.109375 0.421875 -0.421875 1.125q-0.3125 0.703125 -0.984375 2.09375q-0.265625 0.703125 -0.578125 1.3125q-0.3125 0.59375 -0.3125 0.734375q0 0.140625 -0.125 0.421875q-0.125 0.265625 -0.265625 0.6875q-0.125 0.28125 -0.375 0.828125q-0.25 0.53125 -0.515625 1.140625q-0.25 0.59375 -0.421875 1.03125q-0.171875 0.4375 -0.171875 0.5q0 0.140625 -0.015625 0.28125q-0.015625 0.140625 -0.15625 0.28125q-0.140625 0 -0.15625 0.1875q-0.015625 0.1875 -0.015625 0.1875q0 0.34375 -0.5625 1.828125q-0.546875 1.90625 -1.015625 3.515625q-0.46875 1.59375 -0.46875 2.5625q0 0.828125 -0.265625 1.28125q-0.25 0.453125 -0.53125 0.578125zm3.6063232 8.53125q-0.765625 0.140625 -1.046875 0.03125q-0.265625 -0.09375 -0.765625 -0.34375q-0.375 -0.28125 -0.546875 -0.421875q-0.171875 -0.125 -0.25 -0.265625q-0.34375 -0.734375 -0.09375 -1.078125q0.234375 -0.34375 0.703125 -0.4375q0.46875 -0.078125 0.8125 -0.078125q0.25 0.140625 0.8125 0.140625q0.578125 0 0.71875 -0.140625q0 0 0.328125 -0.203125q0.328125 -0.1875 0.890625 -0.28125q1.0625 -0.53125 1.6875 -1.0625q0.625 -0.53125 1.390625 -1.8125q0.3125 -0.5 0.65625 -1.140625q0.34375 -0.640625 0.484375 -1.265625q0.28125 -0.6875 0.640625 -1.859375q0.375 -1.1875 0.640625 -2.34375q0.28125 -1.15625 0.3125 -1.75q0.21875 -1.9375 0.796875 -3.75q0.59375 -1.828125 0.9375 -3.875q0.203125 -0.484375 0.3125 -0.5625q0.109375 -0.09375 0.453125 -0.09375q0.375 0 0.890625 0.234375q0.53125 0.21875 0.765625 0.59375q0.359375 0.65625 0.328125 1.546875q-0.015625 0.875 -0.453125 1.546875q-0.359375 0.484375 -0.625 1.5q-0.25 1.015625 -0.484375 2.546875q-0.1875 0.765625 -0.375 1.796875q-0.1875 1.015625 -0.390625 2.265625q-0.140625 0.34375 -0.390625 1.0625q-0.234375 0.703125 -0.3125 1.109375q-0.03125 0.140625 -0.203125 0.546875q-0.171875 0.421875 -0.34375 0.8125q-0.171875 0.40625 -0.25 0.546875q-1.65625 3.359375 -3.828125 4.84375q-2.15625 1.5 -4.203125 1.640625zm2.90625 -9.8125q-0.375 0 -0.78125 -0.40625q-0.390625 -0.421875 -0.53125 -0.65625q-0.140625 -0.359375 -0.53125 -0.671875q-0.375 -0.3125 -0.65625 -0.75q-0.265625 -0.453125 -0.125 -1.109375q0.125 -0.453125 0.40625 -1.375q0.28125 -0.921875 0.65625 -2.046875q0.390625 -1.125 0.78125 -2.234375q0.40625 -1.109375 0.75 -1.96875q0.34375 -0.875 0.59375 -1.21875q0.203125 -0.28125 0.65625 -0.40625q0.453125 -0.140625 0.890625 -0.078125q0.453125 0.0625 0.671875 0.421875q0.375 0.484375 0.421875 0.953125q0.0625 0.46875 -0.5 1.46875q-0.140625 0.375 -0.34375 0.8125q-0.203125 0.4375 -0.203125 0.578125q-0.046875 0.0625 -0.140625 0.46875q-0.078125 0.390625 -0.1875 0.734375q-0.09375 0.34375 -0.203125 0.34375q-0.125 0.140625 -0.28125 0.734375q-0.15625 0.59375 -0.28125 1.3125q-0.125 0.703125 -0.15625 1.234375q-0.03125 0.53125 0.09375 0.53125q1.078125 0 2.078125 -0.671875q1.015625 -0.671875 2.1875 -2.265625l0.3125 3.25q-0.484375 0.25 -0.90625 0.515625q-0.40625 0.25 -0.75 0.59375q-0.109375 0.140625 -0.5 0.390625q-0.375 0.234375 -0.734375 0.4375q-0.359375 0.1875 -0.46875 0.1875q0 0 -0.09375 0.015625q-0.078125 0.015625 -0.078125 0.15625q0 0.0625 -0.453125 0.265625q-0.453125 0.1875 -0.9375 0.328125q-0.484375 0.125 -0.65625 0.125zm21.626465 2.984375q-1.703125 -0.03125 -2.171875 -0.953125q-0.453125 -0.921875 -0.25 -2.296875q0.203125 -0.671875 0.25 -1.203125q0.0625 -0.53125 0.296875 -0.671875q0 0 0.046875 -0.09375q0.0625 -0.109375 -0.046875 -0.21875q0 -0.453125 0.3125 -1.25q0.078125 -0.203125 0.1875 -0.5q0.125 -0.296875 0.125 -0.296875q-0.171875 -0.28125 0 -0.453125q0.171875 -0.171875 0.359375 -0.328125q0.203125 -0.15625 0.03125 -0.359375q-0.078125 -0.140625 0.15625 -0.640625q0.25 -0.5 0.5625 -1.046875q0.3125 -0.5625 0.421875 -0.875q0 -0.078125 0.171875 -0.484375q0.171875 -0.421875 0.40625 -0.96875q0.25 -0.5625 0.46875 -1.015625q0.234375 -0.453125 0.296875 -0.59375q0.359375 -0.171875 -0.265625 -0.234375q-0.625 -0.0625 -1.328125 -0.03125q-1.0625 -0.03125 -1.71875 -0.09375q-0.640625 -0.078125 -1.046875 -0.328125q-0.421875 -0.484375 -0.109375 -1.15625q0.3125 -0.671875 1.015625 -0.71875q0.4375 -0.03125 0.703125 -0.03125q0.265625 0 0.6875 -0.046875q0.4375 -0.0625 1.375 -0.15625l1.765625 -0.140625l0.109375 -0.59375q0.265625 -0.453125 0.375 -0.578125q0.109375 -0.140625 0.140625 -0.296875q0.03125 -0.15625 0.109375 -0.578125q0 0 0.3125 -0.5625q0.3125 -0.578125 0.375 -1.203125q0.421875 -0.6875 0.734375 -1.265625q0.3125 -0.578125 0.375 -0.703125q0.59375 -0.875 1.15625 -0.9375q0.578125 -0.078125 1.09375 0.265625q0.3125 0.171875 0.59375 0.4375q0.28125 0.265625 0.34375 0.671875q0.0625 0.390625 -0.3125 1.015625q-0.0625 0.140625 -0.203125 0.5q-0.140625 0.359375 -0.21875 0.609375q-0.203125 0.234375 -0.5 0.78125q-0.296875 0.53125 -0.390625 1.078125q-0.28125 0.59375 -0.359375 0.734375q-0.0625 0.140625 -0.203125 0.140625q0.078125 0.34375 0.421875 0.328125q0.34375 -0.015625 1.140625 -0.046875q0.65625 -0.078125 0.921875 -0.15625q0.265625 -0.09375 0.609375 0.078125q0.171875 0.25 0.1875 0.71875q0.015625 0.46875 -0.125 0.90625q-0.140625 0.421875 -0.375 0.53125q-0.171875 0 -0.71875 0.1875q-0.53125 0.1875 -1.328125 0.21875l-1.84375 0.171875l-0.578125 1.1875q-0.25 0.453125 -0.421875 0.8125q-0.171875 0.359375 -0.25 0.4375q0 0.203125 -0.21875 0.625q-0.21875 0.40625 -0.46875 0.890625q-0.203125 0.59375 -0.46875 1.078125q-0.25 0.484375 -0.25 0.484375q-0.140625 0 -0.140625 0.203125q0.171875 0.421875 -0.109375 0.625q-0.171875 0.109375 -0.296875 0.296875q-0.125 0.1875 -0.125 0.1875q0.140625 0.140625 0.046875 0.359375q-0.078125 0.203125 -0.21875 0.4375q0 0.28125 -0.15625 0.8125q-0.15625 0.515625 -0.328125 0.96875q0.03125 0.609375 0.0625 0.796875q0.046875 0.171875 0.1875 0.125q0.171875 0.046875 0.484375 0.015625q0.3125 -0.046875 0.453125 -0.046875q0.515625 -0.234375 -0.484375 2.421875q-0.5625 0.875 -1.125 1.4375q-0.546875 0.578125 -0.71875 0.578125zm9.596252 -3.390625q-0.234375 0.03125 -0.53125 -0.1875q-0.296875 -0.234375 -0.53125 -0.40625q-0.359375 -0.421875 -0.5 -0.734375q-0.125 -0.3125 -0.125 -0.6875q0 -0.375 0.0625 -0.90625q0.171875 -0.546875 0.40625 -1.484375q0.25 -0.9375 0.59375 -2.015625q0.078125 -0.40625 0.265625 -0.96875q0.1875 -0.578125 0.390625 -1.109375q0.21875 -0.546875 0.3125 -0.828125q0.25 -0.859375 0.5 -1.65625q0.265625 -0.796875 0.640625 -1.828125q0.390625 -1.046875 0.953125 -2.65625q0.75 -2.078125 1.125 -3.109375q0.390625 -1.046875 0.5625 -1.484375q0.171875 -0.453125 0.25 -0.734375q0 -0.28125 0.390625 -0.53125q0.40625 -0.265625 0.71875 -0.265625q0.484375 0.03125 0.921875 0.578125q0.453125 0.53125 0.625 1.28125q0.1875 0.734375 -0.234375 1.4375q-0.546875 1.234375 -1.09375 2.78125q-0.53125 1.546875 -1.0625 3.078125q-0.515625 1.515625 -1.03125 2.796875q-0.03125 0.171875 -0.234375 0.59375q-0.1875 0.40625 -0.328125 0.6875q-0.125 0.28125 -0.25 0.53125q-0.125 0.234375 -0.125 0.234375q-0.03125 0.203125 -0.140625 0.59375q-0.09375 0.375 -0.203125 0.65625q-0.03125 0.171875 -0.109375 0.453125q-0.0625 0.265625 -0.09375 0.546875q0.171875 -0.203125 0.484375 -0.5q0.3125 -0.296875 0.796875 -0.75q0.546875 -0.484375 0.953125 -0.828125q0.421875 -0.34375 0.84375 -0.625q0.375 -0.28125 0.703125 -0.515625q0.34375 -0.25 0.34375 -0.25q0.0625 -0.09375 0.140625 -0.234375q0.09375 -0.140625 0.203125 -0.171875q0.0625 -0.078125 0.234375 -0.1875q0.171875 -0.125 0.34375 -0.265625q0.3125 -0.203125 0.65625 -0.34375q0.359375 -0.140625 0.734375 -0.203125q0.390625 -0.078125 0.734375 -0.078125q0.40625 0 0.859375 0.328125q0.453125 0.328125 0.828125 0.921875q0.359375 0.453125 0.4375 1.0625q0.09375 0.59375 0.015625 1.609375q0 0.265625 -0.15625 1.03125q-0.15625 0.765625 -0.21875 1.140625l-0.3125 1.1875l0.28125 0q0.375 -0.140625 0.734375 -0.03125q0.37506104 0.09375 0.37506104 0.09375q0.171875 0.109375 0.328125 0.25q0.15625 0.140625 0.296875 0.203125q0.203125 0.28125 -0.078125 0.796875q-0.28125 0.515625 -0.62506104 0.734375q-0.828125 0.546875 -1.46875 0.6875q-0.640625 0.140625 -1.328125 -0.25q-0.671875 -0.4375 -0.890625 -1.0625q-0.21875 -0.625 -0.1875 -1.703125q0.140625 -1.53125 0.171875 -2.203125q0.03125 -0.671875 -0.34375 -0.671875q-0.421875 0 -1.140625 0.40625q-0.703125 0.390625 -1.578125 1.0625q-0.859375 0.671875 -1.703125 1.5q-0.84375 0.8125 -1.515625 1.640625q-0.40625 0.515625 -0.890625 1.015625q-0.484375 0.5 -1.1875 0.546875zm21.736145 0.625q-1.046875 0.578125 -2.203125 0.671875q-1.15625 0.078125 -2.171875 -0.3125q-1.0 -0.40625 -1.625 -1.203125q-0.34375 -0.4375 -0.65625 -1.421875q-0.3125 -1.0 -0.109375 -2.703125q0.21875 -1.59375 0.984375 -3.21875q0.78125 -1.625 1.921875 -2.875q1.15625 -1.25 2.5 -1.703125q0.484375 -0.171875 0.984375 -0.25q0.515625 -0.09375 1.34375 -0.015625q0.796875 0.09375 1.421875 0.390625q0.625 0.296875 0.90625 0.640625q0.546875 0.5625 0.71875 1.078125q0.171875 0.515625 0.140625 1.0q-0.171875 1.109375 -0.984375 2.390625q-0.8125 1.28125 -2.265625 1.90625q-0.59375 0.25 -1.390625 0.328125q-0.796875 0.0625 -2.1875 -0.21875q-0.828125 -0.171875 -1.09375 0.171875q-0.25 0.34375 -0.21875 1.359375q0.0625 0.609375 0.28125 1.078125q0.234375 0.46875 0.859375 0.640625q0.796875 0.140625 1.140625 0.03125q0.34375 -0.125 0.875 -0.375q0.265625 -0.171875 0.53125 -0.375q0.265625 -0.203125 0.515625 -0.40625q0.265625 -0.21875 0.4375 -0.328125q0.171875 -0.140625 0.34375 -0.171875q0.171875 -0.03125 0.34375 -0.03125q0.390625 0.03125 0.734375 0.421875q0.34375 0.375 0.453125 0.71875q0 0.28125 -0.359375 0.78125q-0.34375 0.5 -0.921875 1.046875q-0.5625 0.53125 -1.25 0.953125zm-0.453125 -7.734375q0.25 -0.109375 0.703125 -0.484375q0.46875 -0.375 0.84375 -0.8125q0.390625 -0.4375 0.390625 -0.671875q0 -0.390625 -0.5 -0.65625q-0.5 -0.28125 -1.296875 -0.109375q-0.28125 0.0625 -0.671875 0.375q-0.375 0.3125 -0.75 0.71875q-0.375 0.390625 -0.65625 0.765625q-0.28125 0.359375 -0.390625 0.5625q0.140625 0.171875 0.5625 0.28125q0.421875 0.09375 0.921875 0.09375q0.5 0 0.84375 -0.0625zm16.977661 8.625q-1.109375 0 -2.03125 -0.5q-0.90625 -0.5 -1.46875 -1.234375q-0.546875 -0.75 -0.546875 -1.453125q0 -0.3125 0.109375 -0.4375q0.125 -0.140625 0.375 -0.140625q0.171875 0.03125 0.40625 0.171875q0.25 0.140625 0.5 0.3125q0.546875 0.375 1.203125 0.484375q0.65625 0.09375 1.171875 -0.09375q0.53125 -0.1875 0.59375 -0.609375q0 -0.140625 -0.140625 -0.578125q-0.125 -0.453125 -0.34375 -0.875q-0.859375 -1.28125 -1.078125 -2.4375q-0.203125 -1.171875 -0.265625 -2.34375q0 -0.484375 0.4375 -1.09375q0.453125 -0.609375 1.125 -1.1875q0.71875 -0.625 1.59375 -1.171875q0.890625 -0.5625 1.75 -0.9375q0.875 -0.390625 1.46875 -0.453125q0.625 0 1.328125 0.046875q0.71875 0.046875 0.953125 0.25q0.484375 0.390625 0.890625 1.265625q0.421875 0.859375 0.421875 1.4375q0 0.28125 -0.296875 0.6875q-0.28125 0.390625 -0.53125 0.875q-0.140625 0.25 -0.25 0.53125q-0.09375 0.265625 -0.234375 0.546875q-0.203125 0.484375 -0.75 0.40625q-0.53125 -0.09375 -0.984375 -0.5625q-0.453125 -0.46875 -0.453125 -1.09375q0 -0.515625 0.3125 -0.890625q0.3125 -0.390625 0.3125 -0.53125q0 -0.0625 -0.0625 -0.09375q-0.0625 -0.046875 -0.25 -0.046875q-0.234375 0 -0.78125 0.203125q-0.53125 0.1875 -1.125 0.53125q-0.578125 0.34375 -1.0625 0.828125q-0.34375 0.390625 -0.578125 0.828125q-0.21875 0.421875 -0.21875 0.90625q0 0.484375 0.1875 1.125q0.1875 0.640625 0.4375 1.0625q0.0625 0.0625 0.21875 0.359375q0.15625 0.296875 0.359375 0.71875q1.28125 2.625 0.4375 3.90625q-0.84375 1.28125 -3.140625 1.28125zm11.957886 -0.6875q-0.28125 0.0625 -0.734375 0.046875q-0.4375 -0.015625 -0.765625 -0.09375q-0.328125 -0.09375 -0.328125 -0.203125q0 -0.171875 -0.171875 -0.171875q0 0 -0.390625 -0.28125q-0.375 -0.296875 -0.6875 -0.4375q-0.3125 -0.484375 -0.40625 -1.125q-0.078125 -0.640625 0.40625 -2.28125q0.03125 -1.109375 0.609375 -2.328125q0.59375 -1.234375 1.5 -2.40625q0.90625 -1.1875 1.875 -2.09375q0.96875 -0.921875 1.734375 -1.375q0.484375 -0.484375 1.296875 -0.515625q0.8125 -0.046875 1.4375 0.296875q0.34375 0.1875 0.75 0.703125q0.421875 0.515625 0.796875 1.09375q0.390625 0.5625 0.609375 1.015625q0.09375 0.375 0.171875 0.671875q0.09375 0.296875 0.125 0.71875q0.046875 0.40625 -0.0625 1.25q-0.171875 0.625 -0.296875 1.265625q-0.125 0.640625 -0.125 1.296875q-0.0625 0.65625 0.15625 0.9375q0.234375 0.265625 0.6875 0.03125q0.40625 -0.203125 0.859375 -0.046875q0.453125 0.15625 0.640625 0.5625q0.1875 0.390625 -0.125 1.0q-0.203125 0.453125 -0.65625 0.875q-0.453125 0.421875 -1.015625 0.6875q-0.546875 0.25 -1.0625 0.25q-0.375 0 -0.765625 -0.296875q-0.375 -0.296875 -0.6875 -0.8125q-0.3125 -0.515625 -0.515625 -1.140625l-0.21875 -0.796875l-1.234375 1.140625q-0.453125 0.484375 -0.890625 0.890625q-0.4375 0.390625 -0.53125 0.5625q-0.171875 0.3125 -0.6875 0.5625q-0.5 0.234375 -1.296875 0.546875zm0.9375 -3.328125q0.59375 -0.34375 1.015625 -0.828125q0.4375 -0.484375 0.84375 -1.09375q0.421875 -0.609375 0.921875 -1.296875q0.578125 -0.796875 0.84375 -1.28125q0.265625 -0.484375 0.390625 -1.046875q-0.125 -0.375 -0.375 -0.8125q-0.25 -0.4375 -0.484375 -0.46875q-0.484375 0 -1.171875 0.59375q-0.671875 0.578125 -1.46875 1.6875q-0.140625 0.1875 -0.34375 0.5625q-0.203125 0.375 -0.40625 0.703125q-0.1875 0.328125 -0.25 0.4375q-0.109375 0.140625 -0.296875 0.640625q-0.1875 0.5 -0.328125 1.015625q-0.140625 0.5 -0.140625 0.703125q0 0.203125 0.046875 0.453125q0.0625 0.234375 0.328125 0.3125q0.28125 0.0625 0.875 -0.28125zm11.990234 2.671875q-0.65625 -0.28125 -1.0625 -0.84375q-0.390625 -0.578125 -0.390625 -1.40625q0 -0.359375 0.140625 -1.1875q0.140625 -0.828125 0.375 -1.859375q0.25 -1.046875 0.484375 -1.953125q0.34375 -0.625 0.46875 -1.25q0.125 -0.625 0.125 -0.75q0.0625 -0.421875 0.234375 -1.109375q0.1875 -0.703125 0.375 -1.265625q0.1875 -0.578125 0.25 -0.640625q0.1875 -0.171875 0.59375 -0.203125q0.421875 -0.046875 0.828125 0.078125q0.421875 0.125 0.5625 0.46875q0.375 0.40625 0.421875 1.25q0.0625 0.828125 -0.3125 1.9375q-0.3125 0.6875 -0.625 1.625q-0.3125 0.9375 -0.546875 1.71875q-0.21875 0.78125 -0.25 1.0625l-1.25 -0.046875q0.578125 -0.546875 1.09375 -1.15625q0.53125 -0.609375 0.78125 -0.953125q0.578125 -0.796875 1.015625 -1.328125q0.4375 -0.546875 1.125 -1.3125q0.6875 -0.796875 1.484375 -1.125q0.796875 -0.328125 1.1875 -0.328125q0.765625 0.140625 1.359375 0.78125q0.609375 0.640625 0.75 1.890625q0 0.484375 -0.09375 1.140625q-0.078125 0.65625 -0.078125 0.65625q0 0 0.5625 -0.6875q0.578125 -0.6875 1.546875 -1.5625q0.65625 -0.484375 1.28125 -0.5625q0.625 -0.09375 1.109375 0.421875q0.90625 0.484375 1.3125 1.703125q0.421875 1.21875 0.28125 2.734375q-0.09375 0.90625 0.015625 1.453125q0.125 0.53125 0.265625 0.9375q0.453125 0.625 0.140625 1.09375q-0.3125 0.46875 -0.546875 0.46875q-0.359375 0 -1.046875 -0.140625q-0.6875 -0.140625 -1.265625 -0.484375q-0.578125 -0.34375 -0.578125 -0.9375l0 -2.140625q0 -0.765625 -0.046875 -1.125q-0.046875 -0.375 -0.1875 -0.46875q-0.375 -0.21875 -0.9375 0.3125q-0.546875 0.515625 -1.203125 1.46875q-0.640625 0.953125 -1.359375 2.125q-0.65625 1.078125 -1.078125 1.3125q-0.40625 0.21875 -0.796875 -0.03125q-0.765625 -0.234375 -0.953125 -0.65625q-0.1875 -0.4375 -0.046875 -1.0q0.171875 -0.59375 0.421875 -1.515625q0.265625 -0.9375 0.5 -1.828125q0.25 -0.890625 0.328125 -1.296875q-0.140625 -0.28125 -0.46875 -0.1875q-0.328125 0.078125 -0.6875 0.328125q-0.40625 0.265625 -0.875 0.578125q-0.46875 0.3125 -1.09375 1.328125q-0.421875 0.65625 -0.9375 1.4375q-0.515625 0.78125 -0.953125 1.40625q-0.4375 0.609375 -0.703125 0.859375q-0.25 0.625 -0.453125 0.78125q-0.203125 0.15625 -0.59375 -0.015625zm20.235107 0.828125q-1.34375 -0.546875 -1.875 -1.796875q-0.515625 -1.25 -0.265625 -3.328125q0.0625 -1.078125 -0.078125 -1.4375q-0.125 -0.375 -0.515625 -0.71875q-0.09375 -0.140625 -0.484375 -0.46875q-0.375 -0.328125 -0.34375 -0.5q-0.03125 -0.234375 0.09375 -0.59375q0.140625 -0.375 0.25 -0.40625q0.25 -0.109375 0.5 -0.25q0.265625 -0.140625 0.40625 -0.0625q0.140625 0 0.421875 0.171875q0.296875 0.171875 0.296875 0.3125q0.140625 0.234375 0.4375 -0.203125q0.296875 -0.453125 0.640625 -1.15625q0.34375 -0.546875 0.859375 -1.03125q0.53125 -0.484375 1.0625 -0.875q0.546875 -0.40625 0.921875 -0.609375q0.171875 -0.078125 0.671875 -0.21875q0.5 -0.140625 1.046875 -0.21875q0.5625 -0.09375 0.84375 -0.0625q0.625 0.046875 1.3125 0.53125q0.703125 0.484375 1.078125 0.890625q0.453125 0.3125 0.609375 0.953125q0.15625 0.640625 0.125 1.34375q-0.03125 0.6875 -0.28125 1.171875q-0.109375 0.34375 -0.734375 0.96875q-0.609375 0.625 -1.359375 1.203125q-0.75 0.5625 -1.265625 0.78125q-0.453125 0.09375 -1.078125 0.15625q-0.625 0.046875 -1.171875 0q-0.53125 -0.0625 -0.734375 -0.15625q-0.234375 -0.140625 -0.390625 0.09375q-0.15625 0.21875 -0.265625 1.1875q-0.03125 0.375 0.015625 0.65625q0.046875 0.28125 0.125 0.421875q0.234375 0.40625 0.46875 0.609375q0.25 0.1875 0.921875 0.109375q0.65625 -0.0625 1.328125 -0.8125q0.671875 -0.75 1.234375 -1.203125q0.140625 -0.234375 0.59375 -0.3125q0.46875 -0.09375 1.140625 0.078125q0.375 0.234375 0.359375 0.765625q-0.015625 0.515625 -0.578125 1.515625q-0.09375 0.25 -0.5625 0.71875q-0.46875 0.46875 -1.015625 0.9375q-0.53125 0.46875 -0.875 0.671875q-1.046875 0.546875 -2.046875 0.515625q-1.0 -0.03125 -1.84375 -0.34375zm2.578125 -7.796875q0.515625 -0.140625 0.875 -0.28125q0.359375 -0.140625 0.703125 -0.453125q0.359375 -0.3125 0.8125 -0.921875q0.125 -0.390625 0.140625 -0.65625q0.03125 -0.28125 -0.046875 -0.359375q-0.28125 -0.28125 -0.609375 -0.1875q-0.328125 0.09375 -0.703125 0.359375q-0.25 0.109375 -0.6875 0.4375q-0.421875 0.328125 -0.828125 0.734375q-0.390625 0.390625 -0.625 0.71875q-0.21875 0.328125 -0.078125 0.40625q-0.0625 0.140625 0.390625 0.171875q0.46875 0.03125 0.65625 0.03125z"
fill-rule="nonzero" />
<path fill="#7c7ce0"
d="m736.979 367.00525l114.703186 0l0 -13.535431l16.5094 27.070862l-16.5094 27.070862l0 -13.535431l-114.703186 0z"
fill-rule="evenodd" />
<path shape-rendering="crispEdges" fill="#b6d7a8"
d="m335.4777 27.61155l30.595795 0l0 22.398949l-30.595795 0l0 -22.398949z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#b6d7a8"
d="m366.0735 27.61155l30.595795 0l0 22.398949l-30.595795 0l0 -22.398949z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#b6d7a8"
d="m396.66928 27.61155l30.595825 0l0 22.398949l-30.595825 0l0 -22.398949z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#b6d7a8"
d="m335.4777 50.010498l30.595795 0l0 22.398949l-30.595795 0l0 -22.398949z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#b6d7a8"
d="m366.0735 50.010498l30.595795 0l0 22.398949l-30.595795 0l0 -22.398949z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#b6d7a8"
d="m396.66928 50.010498l30.595825 0l0 22.398949l-30.595825 0l0 -22.398949z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#b6d7a8"
d="m335.4777 72.40945l30.595795 0l0 22.398949l-30.595795 0l0 -22.398949z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#b6d7a8"
d="m366.0735 72.40945l30.595795 0l0 22.398949l-30.595795 0l0 -22.398949z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#b6d7a8"
d="m396.66928 72.40945l30.595825 0l0 22.398949l-30.595825 0l0 -22.398949z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#b6d7a8"
d="m335.4777 94.808395l30.595795 0l0 22.398956l-30.595795 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#b6d7a8"
d="m366.0735 94.808395l30.595795 0l0 22.398956l-30.595795 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#b6d7a8"
d="m396.66928 94.808395l30.595825 0l0 22.398956l-30.595825 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#6aa84f"
d="m335.4777 117.20735l30.595795 0l0 22.398941l-30.595795 0l0 -22.398941z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#6aa84f"
d="m366.0735 117.20735l30.595795 0l0 22.398941l-30.595795 0l0 -22.398941z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#6aa84f"
d="m396.66928 117.20735l30.595825 0l0 22.398941l-30.595825 0l0 -22.398941z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#b6d7a8"
d="m335.4777 139.6063l30.595795 0l0 22.398956l-30.595795 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#b6d7a8"
d="m366.0735 139.6063l30.595795 0l0 22.398956l-30.595795 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#b6d7a8"
d="m396.66928 139.6063l30.595825 0l0 22.398956l-30.595825 0l0 -22.398956z"
fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m335.4777 27.112862l0 89.5958" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m335.4777 116.708664l0 22.398941" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m335.4777 139.1076l0 23.396332" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m366.0735 27.112862l0 89.5958" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m366.0735 116.708664l0 22.398941" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m366.0735 139.1076l0 23.396332" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m396.66928 27.112862l0 89.5958" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m396.66928 116.708664l0 22.398941" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m396.66928 139.1076l0 23.396332" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m427.2651 27.112862l0 89.5958" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m427.2651 116.708664l0 22.398941" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m427.2651 139.1076l0 23.396332" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m334.979 27.61155l92.78479 0" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m334.979 50.010498l92.78479 0" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m334.979 72.40945l92.78479 0" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m334.979 94.808395l92.78479 0" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m334.979 117.20735l92.78479 0" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m334.979 139.6063l92.78479 0" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m334.979 162.00525l92.78479 0" fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#d5a6bd"
d="m335.4777 313.88977l30.595795 0l0 22.398956l-30.595795 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#d5a6bd"
d="m366.0735 313.88977l30.595795 0l0 22.398956l-30.595795 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#d5a6bd"
d="m396.66928 313.88977l30.595825 0l0 22.398956l-30.595825 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#d5a6bd"
d="m335.4777 336.28873l30.595795 0l0 22.398926l-30.595795 0l0 -22.398926z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#d5a6bd"
d="m366.0735 336.28873l30.595795 0l0 22.398926l-30.595795 0l0 -22.398926z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#d5a6bd"
d="m396.66928 336.28873l30.595825 0l0 22.398926l-30.595825 0l0 -22.398926z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#d5a6bd"
d="m335.4777 358.68765l30.595795 0l0 22.398956l-30.595795 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#d5a6bd"
d="m366.0735 358.68765l30.595795 0l0 22.398956l-30.595795 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#d5a6bd"
d="m396.66928 358.68765l30.595825 0l0 22.398956l-30.595825 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#d5a6bd"
d="m335.4777 381.0866l30.595795 0l0 22.398956l-30.595795 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#d5a6bd"
d="m366.0735 381.0866l30.595795 0l0 22.398956l-30.595795 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#d5a6bd"
d="m396.66928 381.0866l30.595825 0l0 22.398956l-30.595825 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#c27ba0"
d="m335.4777 403.48557l30.595795 0l0 22.398956l-30.595795 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#c27ba0"
d="m366.0735 403.48557l30.595795 0l0 22.398956l-30.595795 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#c27ba0"
d="m396.66928 403.48557l30.595825 0l0 22.398956l-30.595825 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#d5a6bd"
d="m335.4777 425.88452l30.595795 0l0 22.398956l-30.595795 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#d5a6bd"
d="m366.0735 425.88452l30.595795 0l0 22.398956l-30.595795 0l0 -22.398956z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#d5a6bd"
d="m396.66928 425.88452l30.595825 0l0 22.398956l-30.595825 0l0 -22.398956z"
fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m335.4777 313.39108l0 89.595795" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m335.4777 402.98688l0 22.398956" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m335.4777 425.38583l0 23.396332" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m366.0735 313.39108l0 89.595795" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m366.0735 402.98688l0 22.398956" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m366.0735 425.38583l0 23.396332" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m396.66928 313.39108l0 89.595795" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m396.66928 402.98688l0 22.398956" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m396.66928 425.38583l0 23.396332" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m427.2651 313.39108l0 89.595795" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m427.2651 402.98688l0 22.398956" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m427.2651 425.38583l0 23.396332" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m334.979 313.88977l92.78479 0" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m334.979 336.28873l92.78479 0" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m334.979 358.68765l92.78479 0" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m334.979 381.0866l92.78479 0" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m334.979 403.48557l92.78479 0" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m334.979 425.88452l92.78479 0" fill-rule="nonzero" />
<path stroke="#9e9e9e" stroke-width="1.0" stroke-linecap="butt"
d="m334.979 448.28348l92.78479 0" fill-rule="nonzero" />
<path fill="#7c7ce0"
d="m213.92123 148.87953l94.19919 -54.37874l-6.0551147 -10.48819l24.903168 13.591339l-0.68270874 28.361412l-6.0551147 -10.48819l-94.19917 54.378746z"
fill-rule="evenodd" />
<path fill="#7c7ce0"
d="m226.03221 295.8483l94.19917 54.378754l6.0551147 -10.48819l0.68270874 28.36142l-24.903168 13.591339l6.0551147 -10.48819l-94.19917 -54.378754z"
fill-rule="evenodd" />
<path fill="#fce5cd"
d="m29.88197 214.84023l0 0c-1.8811607 -15.205673 4.295059 -30.258377 15.907867 -38.770676c11.612804 -8.512299 26.625553 -8.991333 38.66779 -1.233841l0 0c4.265732 -8.841202 12.073257 -14.94545 21.060959 -16.466324c8.987701 -1.5208588 18.099937 1.7202911 24.58043 8.743057l0 0c3.63385 -8.016006 10.768982 -13.401733 18.873474 -14.246063c8.104507 -0.8443146 16.031372 2.9722748 20.967758 10.095444l0 0c6.5650787 -8.496948 17.0103 -12.074524 26.815933 -9.184677c9.805649 2.8898468 17.21054 11.728058 19.010529 22.690308l0 0c8.04335 2.4131622 14.743271 8.54776 18.36876 16.818802c3.625473 8.271042 3.8208466 17.867157 0.53564453 26.309021l0 0c7.920334 11.338333 9.773102 26.446533 4.8668823 39.686493c-4.9062347 13.239929 -15.834351 22.622498 -28.706192 24.64624c-0.09072876 12.426147 -6.286545 23.828247 -16.19934 29.811432c-9.912781 5.9831543 -21.994614 5.613098 -31.588638 -0.9675598l0 0c-4.0865326 14.882507 -15.588776 25.832886 -29.537323 28.120209c-13.948563 2.287323 -27.842873 -4.498474 -35.680084 -17.42566l0 0c-9.606781 6.3718567 -21.134125 8.207367 -31.981682 5.0925293c-10.847565 -3.1148682 -20.101604 -10.917725 -25.674595 -21.648407l0 0c-9.816925 1.2635498 -19.308535 -4.3308105 -23.76418 -14.006653c-4.4556427 -9.675842 -2.926818 -21.373383 3.8277225 -29.2872l0 0c-8.756951 -5.6690674 -13.225276 -16.918411 -11.074918 -27.881912c2.1503592 -10.963516 10.432122 -19.156815 20.526672 -20.307434z"
fill-rule="evenodd" />
<path fill="#000000" fill-opacity="0.0"
d="m20.236162 263.61563l0 0c4.132433 2.675232 8.906361 3.888794 13.680807 3.4777222m6.253498 39.817078c2.0531883 -0.26428223 4.0657043 -0.8239136 5.9856415 -1.6644592m51.667767 18.219635c-1.4440765 -2.381958 -2.6531372 -4.9272766 -3.6065674 -7.5925903m68.82647 -3.1023865l0 0c0.7450409 -2.7132874 1.227768 -5.5058594 1.4401093 -8.331116m46.34607 -20.511719c0.096588135 -13.229584 -6.7349396 -25.342712 -17.560226 -31.13646m41.399292 -33.194885c-1.7531128 4.5049286 -4.4294586 8.501175 -7.819153 11.67543m-11.083603 -54.80443l0 0c0.29870605 1.8191833 0.43695068 3.6657257 0.41281128 5.5139923m-46.238495 -19.019073l0 0c-1.6376495 2.1195679 -2.9868774 4.488159 -4.0056305 7.032013m-35.835968 -2.882309l0 0c-0.8727417 1.9251862 -1.5244446 3.9623566 -1.940094 6.0646057m-43.701965 1.6588135l0 0c2.5481339 1.6414795 4.9054947 3.6171875 7.0203094 5.8836975m-61.595264 34.121033l0 0c0.25927162 2.0957336 0.6689358 4.165634 1.2251568 6.190323"
fill-rule="evenodd" />
<path stroke="#000000" stroke-width="1.0"
stroke-linejoin="round" stroke-linecap="butt"
d="m29.88197 214.84023l0 0c-1.8811607 -15.205673 4.295059 -30.258377 15.907867 -38.770676c11.612804 -8.512299 26.625553 -8.991333 38.66779 -1.233841l0 0c4.265732 -8.841202 12.073257 -14.94545 21.060959 -16.466324c8.987701 -1.5208588 18.099937 1.7202911 24.58043 8.743057l0 0c3.63385 -8.016006 10.768982 -13.401733 18.873474 -14.246063c8.104507 -0.8443146 16.031372 2.9722748 20.967758 10.095444l0 0c6.5650787 -8.496948 17.0103 -12.074524 26.815933 -9.184677c9.805649 2.8898468 17.21054 11.728058 19.010529 22.690308l0 0c8.04335 2.4131622 14.743271 8.54776 18.36876 16.818802c3.625473 8.271042 3.8208466 17.867157 0.53564453 26.309021l0 0c7.920334 11.338333 9.773102 26.446533 4.8668823 39.686493c-4.9062347 13.239929 -15.834351 22.622498 -28.706192 24.64624c-0.09072876 12.426147 -6.286545 23.828247 -16.19934 29.811432c-9.912781 5.9831543 -21.994614 5.613098 -31.588638 -0.9675598l0 0c-4.0865326 14.882507 -15.588776 25.832886 -29.537323 28.120209c-13.948563 2.287323 -27.842873 -4.498474 -35.680084 -17.42566l0 0c-9.606781 6.3718567 -21.134125 8.207367 -31.981682 5.0925293c-10.847565 -3.1148682 -20.101604 -10.917725 -25.674595 -21.648407l0 0c-9.816925 1.2635498 -19.308535 -4.3308105 -23.76418 -14.006653c-4.4556427 -9.675842 -2.926818 -21.373383 3.8277225 -29.2872l0 0c-8.756951 -5.6690674 -13.225276 -16.918411 -11.074918 -27.881912c2.1503592 -10.963516 10.432122 -19.156815 20.526672 -20.307434z"
fill-rule="evenodd" />
<path stroke="#000000" stroke-width="1.0"
stroke-linejoin="round" stroke-linecap="butt"
d="m20.236162 263.61563l0 0c4.132433 2.675232 8.906361 3.888794 13.680807 3.4777222m6.253498 39.817078c2.0531883 -0.26428223 4.0657043 -0.8239136 5.9856415 -1.6644592m51.667767 18.219635c-1.4440765 -2.381958 -2.6531372 -4.9272766 -3.6065674 -7.5925903m68.82647 -3.1023865l0 0c0.7450409 -2.7132874 1.227768 -5.5058594 1.4401093 -8.331116m46.34607 -20.511719c0.096588135 -13.229584 -6.7349396 -25.342712 -17.560226 -31.13646m41.399292 -33.194885c-1.7531128 4.5049286 -4.4294586 8.501175 -7.819153 11.67543m-11.083603 -54.80443l0 0c0.29870605 1.8191833 0.43695068 3.6657257 0.41281128 5.5139923m-46.238495 -19.019073l0 0c-1.6376495 2.1195679 -2.9868774 4.488159 -4.0056305 7.032013m-35.835968 -2.882309l0 0c-0.8727417 1.9251862 -1.5244446 3.9623566 -1.940094 6.0646057m-43.701965 1.6588135l0 0c2.5481339 1.6414795 4.9054947 3.6171875 7.0203094 5.8836975m-61.595264 34.121033l0 0c0.25927162 2.0957336 0.6689358 4.165634 1.2251568 6.190323"
fill-rule="evenodd" />
<path fill="#000000"
d="m46.73521 236.81306q-1.03125 0 -2.46875 -0.359375q-0.265625 -0.0625 -0.453125 -0.296875q-0.171875 -0.25 -0.171875 -0.546875q0 -0.140625 0.015625 -0.203125l2.671875 -12.46875q0.078125 -0.453125 0.453125 -0.75q0.375 -0.296875 0.828125 -0.296875q2.46875 0 4.328125 0.703125q1.875 0.6875 2.921875 2.046875q1.046875 1.359375 1.046875 3.328125q0 2.578125 -1.171875 4.59375q-1.171875 2.015625 -3.265625 3.140625q-2.09375 1.109375 -4.734375 1.109375zm0.46875 -2.078125q1.953125 0 3.421875 -0.765625q1.484375 -0.78125 2.28125 -2.1875q0.796875 -1.421875 0.796875 -3.28125q0 -2.125 -1.5 -3.203125q-1.484375 -1.078125 -3.984375 -1.28125l-2.234375 10.59375q0.6875 0.125 1.21875 0.125zm12.960831 1.890625q-1.484375 0 -2.375 -0.90625q-0.890625 -0.921875 -0.890625 -2.484375q0 -1.703125 0.921875 -3.390625q0.9375 -1.703125 2.453125 -2.78125q1.515625 -1.078125 3.15625 -1.078125q0.9843788 0 1.7343788 0.265625q0.765625 0.265625 1.484375 0.65625l-1.875 8.875q-0.109375 0.484375 -0.453125 0.765625q-0.3281288 0.296875 -0.7812538 0.296875q-0.359375 0 -0.59375 -0.203125q-0.21875 -0.203125 -0.21875 -0.546875q0 -0.140625 0.03125 -0.234375l0.0625 -0.296875q-0.625 0.53125 -1.328125 0.796875q-0.6875 0.265625 -1.328125 0.265625zm0.515625 -1.953125q0.65625 0 1.28125 -0.265625q0.640625 -0.28125 1.15625 -0.640625l1.2343788 -5.5625q-0.5781288 -0.25 -1.2812538 -0.25q-1.0 0 -1.9375 0.765625q-0.9375 0.765625 -1.546875 1.921875q-0.59375 1.140625 -0.59375 2.234375q0 0.953125 0.421875 1.375q0.4375 0.421875 1.265625 0.421875zm8.959736 1.984375q-0.40625 0 -0.65625 -0.296875q-0.234375 -0.3125 -0.15625 -0.71875l1.625 -7.6875l-1.796875 0q-0.375 0 -0.609375 -0.234375q-0.21875 -0.234375 -0.21875 -0.578125q0 -0.453125 0.40625 -0.796875q0.40625 -0.34375 0.921875 -0.34375l1.75 0l0.484375 -2.3125q0.078125 -0.40625 0.453125 -0.71875q0.375 -0.328125 0.78125 -0.328125q0.390625 0 0.609375 0.25q0.21875 0.234375 0.21875 0.578125q0 0.15625 -0.015625 0.21875l-0.484375 2.3125l1.8125 0q0.359375 0 0.578125 0.21875q0.21875 0.21875 0.21875 0.5625q0 0.46875 -0.390625 0.828125q-0.390625 0.34375 -0.859375 0.34375l-1.8125 0l-1.625 7.6875q-0.078125 0.4375 -0.453125 0.734375q-0.375 0.28125 -0.78125 0.28125zm8.568314 -0.03125q-1.484375 0 -2.375 -0.90625q-0.890625 -0.921875 -0.890625 -2.484375q0 -1.703125 0.921875 -3.390625q0.9375 -1.703125 2.453125 -2.78125q1.515625 -1.078125 3.15625 -1.078125q0.984375 0 1.734375 0.265625q0.765625 0.265625 1.484375 0.65625l-1.875 8.875q-0.109375 0.484375 -0.453125 0.765625q-0.328125 0.296875 -0.78125 0.296875q-0.359375 0 -0.59375 -0.203125q-0.21875 -0.203125 -0.21875 -0.546875q0 -0.140625 0.03125 -0.234375l0.0625 -0.296875q-0.625 0.53125 -1.328125 0.796875q-0.6875 0.265625 -1.328125 0.265625zm0.515625 -1.953125q0.65625 0 1.28125 -0.265625q0.640625 -0.28125 1.15625 -0.640625l1.234375 -5.5625q-0.578125 -0.25 -1.28125 -0.25q-1.0 0 -1.9375 0.765625q-0.9375 0.765625 -1.546875 1.921875q-0.59375 1.140625 -0.59375 2.234375q0 0.953125 0.421875 1.375q0.4375 0.421875 1.265625 0.421875zm8.303482 -2.515625q-0.34375 0 -0.578125 -0.234375q-0.234375 -0.234375 -0.234375 -0.578125q0 -0.46875 0.40625 -0.828125q0.40625 -0.375 0.828125 -0.375l4.796875 0q0.359375 0 0.578125 0.234375q0.234375 0.21875 0.234375 0.578125q0 0.453125 -0.40625 0.828125q-0.390625 0.375 -0.828125 0.375l-4.796875 0zm10.004486 8.0625q-1.265625 0 -2.453125 -0.3125q-0.28125 -0.0625 -0.453125 -0.28125q-0.15625 -0.203125 -0.15625 -0.5q0 -0.46875 0.390625 -0.859375q0.40625 -0.375 0.875 -0.375q0.296875 0 0.78125 0.109375q0.78125 0.15625 1.40625 0.15625q1.140625 0 1.84375 -0.5625q0.703125 -0.546875 1.046875 -1.71875l0.125 -0.421875q-1.1875 0.984375 -2.453125 0.984375q-1.515625 0 -2.46875 -0.96875q-0.9375 -0.96875 -0.9375 -2.6875q0 -1.671875 0.921875 -3.265625q0.921875 -1.59375 2.40625 -2.5625q1.484375 -0.984375 3.0625 -0.984375q0.875 0 1.6875 0.4375q0.140625 -0.296875 0.40625 -0.46875q0.28125 -0.1875 0.578125 -0.1875q0.46875 0 0.6875 0.1875q0.21875 0.171875 0.21875 0.53125q0 0.265625 -0.03125 0.40625l-2.09375 8.921875q-0.515625 2.171875 -1.953125 3.296875q-1.4375 1.125 -3.4375 1.125zm1.234375 -5.78125q1.140625 0 2.0 -0.890625q0.875 -0.890625 1.234375 -2.578125l0.5625 -2.578125q-0.265625 -0.171875 -0.671875 -0.265625q-0.40625 -0.109375 -0.78125 -0.109375q-0.984375 0 -1.9375 0.59375q-0.9375 0.59375 -1.53125 1.59375q-0.578125 1.0 -0.578125 2.15625q0 0.984375 0.4375 1.53125q0.453125 0.546875 1.265625 0.546875zm11.682938 2.1875q-1.96875 0 -3.140625 -1.0625q-1.171875 -1.0625 -1.171875 -2.9375q0 -1.65625 0.859375 -3.1875q0.859375 -1.53125 2.265625 -2.484375q1.40625 -0.96875 2.984375 -0.96875q1.171875 0 1.9375 0.4375q0.78125 0.4375 1.140625 1.078125q0.375 0.640625 0.375 1.234375q0 0.453125 -0.21875 0.765625q-0.203125 0.296875 -0.65625 0.53125l-6.421875 3.296875q0.078125 0.65625 0.765625 1.015625q0.6875 0.359375 1.53125 0.359375q1.390625 0 2.453125 -0.90625q0.4375 -0.34375 0.796875 -0.34375q0.359375 0 0.578125 0.203125q0.234375 0.203125 0.234375 0.515625q0 0.296875 -0.109375 0.515625q-0.109375 0.21875 -0.390625 0.453125q-0.671875 0.609375 -1.765625 1.046875q-1.09375 0.4375 -2.046875 0.4375zm3.171875 -7.828125q-0.140625 -0.46875 -0.578125 -0.703125q-0.421875 -0.234375 -1.0625 -0.234375q-1.359375 0 -2.53125 1.171875q-0.5625 0.5625 -0.890625 1.265625q-0.328125 0.703125 -0.375 1.25l5.4375 -2.75zm3.7434845 7.828125q-0.40625 0 -0.65625 -0.296875q-0.25 -0.296875 -0.15625 -0.6875l1.828125 -8.65625q0.09375 -0.421875 0.46875 -0.71875q0.390625 -0.296875 0.8125 -0.296875q0.34375 0 0.5625 0.21875q0.234375 0.21875 0.234375 0.5625q0 0.15625 -0.03125 0.234375l-0.34375 1.625q0.875 -1.28125 1.9375 -1.953125q1.0625 -0.671875 2.140625 -0.671875q1.234375 0 1.8125 0.609375q0.59375 0.609375 0.59375 2.03125q0 1.328125 -0.421875 3.25l-0.8125 3.734375q-0.078125 0.40625 -0.421875 0.71875q-0.34375 0.296875 -0.765625 0.296875q-0.40625 0 -0.65625 -0.296875q-0.25 -0.3125 -0.15625 -0.703125l0.75 -3.546875q0.40625 -1.890625 0.40625 -2.828125q0 -0.734375 -0.234375 -1.0q-0.234375 -0.28125 -0.703125 -0.28125q-0.640625 0 -1.40625 0.546875q-0.75 0.546875 -1.515625 1.546875q-0.671875 0.875 -1.046875 1.703125q-0.375 0.828125 -0.625 2.0l-0.375 1.796875q-0.09375 0.46875 -0.421875 0.765625q-0.328125 0.296875 -0.796875 0.296875zm15.159348 0q-1.96875 0 -3.140625 -1.0625q-1.171875 -1.0625 -1.171875 -2.9375q0 -1.65625 0.859375 -3.1875q0.859375 -1.53125 2.265625 -2.484375q1.40625 -0.96875 2.984375 -0.96875q1.171875 0 1.9375 0.4375q0.78125 0.4375 1.140625 1.078125q0.375 0.640625 0.375 1.234375q0 0.453125 -0.21875 0.765625q-0.203125 0.296875 -0.65625 0.53125l-6.421875 3.296875q0.078125 0.65625 0.765625 1.015625q0.6875 0.359375 1.53125 0.359375q1.390625 0 2.453125 -0.90625q0.4375 -0.34375 0.796875 -0.34375q0.359375 0 0.578125 0.203125q0.234375 0.203125 0.234375 0.515625q0 0.296875 -0.109375 0.515625q-0.109375 0.21875 -0.390625 0.453125q-0.671875 0.609375 -1.765625 1.046875q-1.09375 0.4375 -2.046875 0.4375zm3.171875 -7.828125q-0.140625 -0.46875 -0.578125 -0.703125q-0.421875 -0.234375 -1.0625 -0.234375q-1.359375 0 -2.53125 1.171875q-0.5625 0.5625 -0.890625 1.265625q-0.328125 0.703125 -0.375 1.25l5.4375 -2.75zm9.9466095 -2.859375q0.78125 0 1.234375 0.390625q0.453125 0.375 0.453125 0.9375q0 0.25 -0.078125 0.6875q-0.078125 0.421875 -0.15625 0.71875q-0.109375 0.390625 -0.4375 0.671875q-0.328125 0.265625 -0.765625 0.265625q-0.375 0 -0.59375 -0.1875q-0.21875 -0.1875 -0.21875 -0.5q0 -0.140625 0.078125 -0.375q0.078125 -0.234375 0.078125 -0.328125q0 -0.25 -0.3125 -0.25q-0.453125 0 -1.15625 0.453125q-0.6875 0.4375 -1.171875 0.953125q-0.703125 0.703125 -0.9375 1.921875l-0.96875 4.34375q-0.078125 0.390625 -0.453125 0.6875q-0.375 0.296875 -0.765625 0.296875q-0.390625 0 -0.671875 -0.296875q-0.265625 -0.3125 -0.171875 -0.6875l1.828125 -8.625q0.09375 -0.4375 0.46875 -0.734375q0.390625 -0.296875 0.8125 -0.296875q0.46875 0 0.671875 0.28125q0.203125 0.28125 0.140625 0.65625l-0.109375 0.578125q0.671875 -0.75 1.5 -1.15625q0.84375 -0.40625 1.703125 -0.40625zm4.1740265 10.6875q-1.484375 0 -2.375 -0.90625q-0.890625 -0.921875 -0.890625 -2.484375q0 -1.703125 0.921875 -3.390625q0.9375 -1.703125 2.453125 -2.78125q1.515625 -1.078125 3.15625 -1.078125q0.984375 0 1.734375 0.265625q0.765625 0.265625 1.484375 0.65625l-1.875 8.875q-0.109375 0.484375 -0.453125 0.765625q-0.328125 0.296875 -0.78125 0.296875q-0.359375 0 -0.59375 -0.203125q-0.21875 -0.203125 -0.21875 -0.546875q0 -0.140625 0.03125 -0.234375l0.0625 -0.296875q-0.625 0.53125 -1.328125 0.796875q-0.6875 0.265625 -1.328125 0.265625zm0.515625 -1.953125q0.65625 0 1.28125 -0.265625q0.640625 -0.28125 1.15625 -0.640625l1.234375 -5.5625q-0.578125 -0.25 -1.28125 -0.25q-1.0 0 -1.9375 0.765625q-0.9375 0.765625 -1.546875 1.921875q-0.59375 1.140625 -0.59375 2.234375q0 0.953125 0.421875 1.375q0.4375 0.421875 1.265625 0.421875zm8.959732 1.984375q-0.40625 0 -0.65625 -0.296875q-0.234375 -0.3125 -0.15625 -0.71875l1.625 -7.6875l-1.796875 0q-0.375 0 -0.609375 -0.234375q-0.21875 -0.234375 -0.21875 -0.578125q0 -0.453125 0.40625 -0.796875q0.40625 -0.34375 0.921875 -0.34375l1.75 0l0.484375 -2.3125q0.078125 -0.40625 0.453125 -0.71875q0.375 -0.328125 0.78125 -0.328125q0.390625 0 0.609375 0.25q0.21875 0.234375 0.21875 0.578125q0 0.15625 -0.015625 0.21875l-0.484375 2.3125l1.8125 0q0.359375 0 0.578125 0.21875q0.21875 0.21875 0.21875 0.5625q0 0.46875 -0.390625 0.828125q-0.390625 0.34375 -0.859375 0.34375l-1.8125 0l-1.625 7.6875q-0.078125 0.4375 -0.453125 0.734375q-0.375 0.28125 -0.78125 0.28125zm8.73027 -11.5625q-0.34375 0 -0.578125 -0.21875q-0.234375 -0.234375 -0.234375 -0.578125q0 -0.078125 0.03125 -0.25l0.203125 -0.796875q0.078125 -0.40625 0.453125 -0.703125q0.390625 -0.296875 0.78125 -0.296875q0.34375 0 0.578125 0.25q0.234375 0.25 0.234375 0.578125q0 0.140625 -0.03125 0.203125l-0.203125 0.8125q-0.109375 0.421875 -0.46875 0.71875q-0.34375 0.28125 -0.765625 0.28125zm-2.546875 11.53125q-0.359375 0 -0.609375 -0.234375q-0.25 -0.234375 -0.25 -0.59375q0 -0.125 0.03125 -0.21875l1.890625 -8.484375q0.09375 -0.421875 0.46875 -0.734375q0.390625 -0.3125 0.8125 -0.3125q0.390625 0 0.609375 0.234375q0.21875 0.21875 0.21875 0.578125q0 0.15625 -0.015625 0.234375l-1.890625 8.484375q-0.09375 0.4375 -0.46875 0.75q-0.375 0.296875 -0.796875 0.296875zm5.1671295 0q-0.40625 0 -0.65625 -0.296875q-0.25 -0.296875 -0.15625 -0.6875l1.828125 -8.65625q0.09375 -0.421875 0.46875 -0.71875q0.390625 -0.296875 0.8125 -0.296875q0.34375 0 0.5625 0.21875q0.234375 0.21875 0.234375 0.5625q0 0.15625 -0.03125 0.234375l-0.34375 1.625q0.875 -1.28125 1.9375 -1.953125q1.0625 -0.671875 2.140625 -0.671875q1.234375 0 1.8125 0.609375q0.59375 0.609375 0.59375 2.03125q0 1.328125 -0.421875 3.25l-0.8125 3.734375q-0.078125 0.40625 -0.421875 0.71875q-0.34375 0.296875 -0.765625 0.296875q-0.40625 0 -0.65625 -0.296875q-0.25 -0.3125 -0.15625 -0.703125l0.75 -3.546875q0.40625 -1.890625 0.40625 -2.828125q0 -0.734375 -0.234375 -1.0q-0.234375 -0.28125 -0.703125 -0.28125q-0.640625 0 -1.40625 0.546875q-0.75 0.546875 -1.515625 1.546875q-0.671875 0.875 -1.046875 1.703125q-0.375 0.828125 -0.625 2.0l-0.375 1.796875q-0.09375 0.46875 -0.421875 0.765625q-0.328125 0.296875 -0.796875 0.296875zm13.268723 3.59375q-1.265625 0 -2.453125 -0.3125q-0.28125 -0.0625 -0.453125 -0.28125q-0.15625 -0.203125 -0.15625 -0.5q0 -0.46875 0.390625 -0.859375q0.40625 -0.375 0.875 -0.375q0.296875 0 0.78125 0.109375q0.78125 0.15625 1.40625 0.15625q1.140625 0 1.84375 -0.5625q0.703125 -0.546875 1.046875 -1.71875l0.125 -0.421875q-1.1875 0.984375 -2.453125 0.984375q-1.515625 0 -2.46875 -0.96875q-0.9375 -0.96875 -0.9375 -2.6875q0 -1.671875 0.921875 -3.265625q0.921875 -1.59375 2.40625 -2.5625q1.484375 -0.984375 3.0625 -0.984375q0.875 0 1.6875 0.4375q0.140625 -0.296875 0.40625 -0.46875q0.28125 -0.1875 0.578125 -0.1875q0.46875 0 0.6875 0.1875q0.21875 0.171875 0.21875 0.53125q0 0.265625 -0.03125 0.40625l-2.09375 8.921875q-0.515625 2.171875 -1.953125 3.296875q-1.4375 1.125 -3.4375 1.125zm1.234375 -5.78125q1.140625 0 2.0 -0.890625q0.875 -0.890625 1.234375 -2.578125l0.5625 -2.578125q-0.265625 -0.171875 -0.671875 -0.265625q-0.40625 -0.109375 -0.78125 -0.109375q-0.984375 0 -1.9375 0.59375q-0.9375 0.59375 -1.53125 1.59375q-0.578125 1.0 -0.578125 2.15625q0 0.984375 0.4375 1.53125q0.453125 0.546875 1.265625 0.546875z"
fill-rule="nonzero" />
<path fill="#000000"
d="m59.146053 262.60995q-1.328125 0 -2.15625 -0.890625q-0.8125 -0.90625 -0.8125 -2.53125q0 -1.765625 0.765625 -3.4218903q0.765625 -1.671875 2.125 -2.71875q1.375 -1.046875 3.0625 -1.046875q1.203125 0 2.1250038 0.640625l0.859375 -3.453125q0.109375 -0.421875 0.46875 -0.734375q0.375 -0.3125 0.8125 -0.3125q0.375 0 0.578125 0.25q0.203125 0.25 0.234375 0.59375q0 0.078125 -0.046875 0.25l-2.921875 12.406265q-0.109375 0.4375 -0.4687538 0.71875q-0.359375 0.265625 -0.796875 0.265625q-0.375 0 -0.59375 -0.203125q-0.21875 -0.21875 -0.21875 -0.5625q0 -0.171875 0.03125 -0.25l0.109375 -0.4375q-0.625 0.640625 -1.46875 1.046875q-0.84375 0.390625 -1.6875 0.390625zm0.5625 -1.921875q0.75 0 1.4375 -0.328125q0.703125 -0.328125 1.171875 -0.828125q0.46875 -0.5 0.59375 -1.015625l0.921875 -3.9218903q-0.34375 -0.265625 -0.890625 -0.46875q-0.546875 -0.203125 -1.09375 -0.203125q-1.09375 0 -1.921875 0.734375q-0.8125 0.734375 -1.234375 1.8906403q-0.421875 1.15625 -0.421875 2.328125q0 0.890625 0.375 1.359375q0.390625 0.453125 1.0625 0.453125zm11.019733 -9.593765q-0.34375 0 -0.578125 -0.21875q-0.234375 -0.234375 -0.234375 -0.578125q0 -0.078125 0.03125 -0.25l0.203125 -0.796875q0.078125 -0.40625 0.453125 -0.703125q0.390625 -0.296875 0.78125 -0.296875q0.34375 0 0.578125 0.25q0.234375 0.25 0.234375 0.578125q0 0.140625 -0.03125 0.203125l-0.203125 0.8125q-0.109375 0.421875 -0.46875 0.71875q-0.34375 0.28125 -0.765625 0.28125zm-2.546875 11.531265q-0.359375 0 -0.609375 -0.234375q-0.25 -0.234375 -0.25 -0.59375q0 -0.125 0.03125 -0.21875l1.890625 -8.48439q0.09375 -0.421875 0.46875 -0.734375q0.390625 -0.3125 0.8125 -0.3125q0.390625 0 0.609375 0.234375q0.21875 0.21875 0.21875 0.578125q0 0.15625 -0.015625 0.234375l-1.890625 8.48439q-0.09375 0.4375 -0.46875 0.75q-0.375 0.296875 -0.796875 0.296875zm7.3546295 0q-1.046875 0 -1.859375 -0.40625q-0.796875 -0.421875 -1.234375 -1.0625q-0.4375 -0.640625 -0.4375 -1.28125q0 -0.46875 0.375 -0.84375q0.390625 -0.375 0.859375 -0.375q0.359375 0 0.546875 0.203125q0.203125 0.1875 0.234375 0.5q0.078125 0.671875 0.546875 1.0625q0.484375 0.375 1.3125 0.375q0.859375 0 1.4375 -0.453125q0.59375 -0.453125 0.59375 -1.0q0 -0.453125 -0.375 -0.71875q-0.359375 -0.265625 -1.328125 -0.578125q-1.40625 -0.46875 -2.046875 -1.109375q-0.640625 -0.640625 -0.640625 -1.6093903q0 -0.953125 0.59375 -1.71875q0.59375 -0.765625 1.578125 -1.1875q0.984375 -0.4375 2.125 -0.4375q1.578125 0 2.375 0.5625q0.8125 0.5625 0.8125 1.640625q0 0.828125 -0.40625 1.328125q-0.40625 0.50001526 -1.0 0.50001526q-0.390625 0 -0.625 -0.20314026q-0.21875 -0.21875 -0.21875 -0.5625q0 -0.15625 0.125 -0.390625q0.140625 -0.234375 0.140625 -0.40625q0 -0.671875 -1.5 -0.671875q-0.984375 0 -1.484375 0.328125q-0.5 0.328125 -0.5 0.8125q0 0.4375 0.375 0.71875q0.375 0.265625 1.484375 0.65626526q1.390625 0.46875 1.984375 1.03125q0.59375 0.546875 0.59375 1.53125q0 0.96875 -0.609375 1.84375q-0.609375 0.875 -1.640625 1.40625q-1.015625 0.515625 -2.1875 0.515625zm7.9394836 0.03125q-0.40625 0 -0.65625 -0.296875q-0.234375 -0.3125 -0.15625 -0.71875l1.625 -7.6875153l-1.796875 0q-0.375 0 -0.609375 -0.234375q-0.21875 -0.234375 -0.21875 -0.578125q0 -0.453125 0.40625 -0.796875q0.40625 -0.34375 0.921875 -0.34375l1.75 0l0.484375 -2.3125q0.078125 -0.40625 0.453125 -0.71875q0.375 -0.328125 0.78125 -0.328125q0.390625 0 0.609375 0.25q0.21875 0.234375 0.21875 0.578125q0 0.15625 -0.015625 0.21875l-0.484375 2.3125l1.8125 0q0.359375 0 0.578125 0.21875q0.21875 0.21875 0.21875 0.5625q0 0.46875 -0.390625 0.828125q-0.390625 0.34375 -0.859375 0.34375l-1.8125 0l-1.625 7.6875153q-0.078125 0.4375 -0.453125 0.734375q-0.375 0.28125 -0.78125 0.28125zm12.370895 -10.718765q0.78125 0 1.234375 0.390625q0.453125 0.375 0.453125 0.9375q0 0.25 -0.078125 0.6875q-0.078125 0.421875 -0.15625 0.71875q-0.109375 0.390625 -0.4375 0.671875q-0.328125 0.265625 -0.765625 0.265625q-0.375 0 -0.59375 -0.1875q-0.21875 -0.1875 -0.21875 -0.5q0 -0.140625 0.078125 -0.375q0.078125 -0.234375 0.078125 -0.328125q0 -0.25 -0.3125 -0.25q-0.453125 0 -1.15625 0.453125q-0.6875 0.4375 -1.171875 0.953125q-0.703125 0.70314026 -0.9375 1.9218903l-0.96875 4.34375q-0.078125 0.390625 -0.453125 0.6875q-0.375 0.296875 -0.765625 0.296875q-0.390625 0 -0.671875 -0.296875q-0.265625 -0.3125 -0.171875 -0.6875l1.828125 -8.625015q0.09375 -0.4375 0.46875 -0.734375q0.390625 -0.296875 0.8125 -0.296875q0.46875 0 0.671875 0.28125q0.203125 0.28125 0.140625 0.65625l-0.109375 0.578125q0.671875 -0.75 1.5 -1.15625q0.84375 -0.40625 1.703125 -0.40625zm4.954651 -0.84375q-0.34375 0 -0.578125 -0.21875q-0.234375 -0.234375 -0.234375 -0.578125q0 -0.078125 0.03125 -0.25l0.203125 -0.796875q0.078125 -0.40625 0.453125 -0.703125q0.390625 -0.296875 0.78125 -0.296875q0.34375 0 0.578125 0.25q0.234375 0.25 0.234375 0.578125q0 0.140625 -0.03125 0.203125l-0.203125 0.8125q-0.109375 0.421875 -0.46875 0.71875q-0.34375 0.28125 -0.765625 0.28125zm-2.546875 11.531265q-0.359375 0 -0.609375 -0.234375q-0.25 -0.234375 -0.25 -0.59375q0 -0.125 0.03125 -0.21875l1.890625 -8.48439q0.09375 -0.421875 0.46875 -0.734375q0.390625 -0.3125 0.8125 -0.3125q0.390625 0 0.609375 0.234375q0.21875 0.21875 0.21875 0.578125q0 0.15625 -0.015625 0.234375l-1.890625 8.48439q-0.09375 0.4375 -0.46875 0.75q-0.375 0.296875 -0.796875 0.296875zm5.1358795 -0.015625q-0.40625 0 -0.640625 -0.28125q-0.234375 -0.296875 -0.140625 -0.6875l2.640625 -12.39064q0.09375 -0.421875 0.4375 -0.6875q0.359375 -0.265625 0.78125 -0.265625q0.421875 0 0.65625 0.296875q0.234375 0.296875 0.15625 0.65625l-0.875 4.078125q0.71875 -0.578125 1.5625 -0.875q0.84375 -0.296875 1.703125 -0.296875q1.515625 0 2.25 0.90625q0.75 0.890625 0.75 2.421875q0 1.0468903 -0.375 2.2656403q-0.375 1.203125 -0.9375 2.125q-0.78125 1.21875 -1.9375 1.921875q-1.140625 0.703125 -2.53125 0.703125q-0.71875 0 -1.390625 -0.203125q-0.671875 -0.21875 -1.03125 -0.484375l-0.046875 0.171875q-0.0625 0.28125 -0.359375 0.453125q-0.296875 0.171875 -0.671875 0.171875zm3.53125 -2.03125q0.9375 0 1.609375 -0.390625q0.671875 -0.390625 1.171875 -1.1875q0.40625 -0.640625 0.65625 -1.515625q0.265625 -0.875 0.265625 -1.5468903q0 -1.859375 -1.625 -1.859375q-1.109375 0 -2.078125 0.703125q-0.96875 0.703125 -1.171875 1.6562653l-0.796875 3.6875q1.0 0.453125 1.96875 0.453125zm9.562393 2.109375q-1.171875 0 -1.765625 -0.59375q-0.578125 -0.59375 -0.578125 -2.0q0 -1.328125 0.4375 -3.359375q0.4375 -2.0468903 1.0 -3.6875153q0.359375 -1.078125 1.234375 -1.078125q0.40625 0 0.609375 0.1875q0.21875 0.1875 0.21875 0.53125q0 0.109375 -0.078125 0.453125q-0.5625 1.78125 -0.953125 3.4062653q-0.390625 1.609375 -0.390625 2.8125q0 0.734375 0.203125 1.078125q0.21875 0.328125 0.6875 0.328125q0.75 0 1.53125 -0.34375q0.796875 -0.359375 1.15625 -0.625q0.28125 -0.203125 0.375 -0.359375q0.109375 -0.15625 0.15625 -0.40625l1.34375 -6.0937653q0.09375 -0.421875 0.4375 -0.6875q0.359375 -0.28125 0.78125 -0.28125q0.421875 0 0.65625 0.296875q0.234375 0.28125 0.15625 0.671875l-1.84375 8.67189q-0.078125 0.421875 -0.453125 0.71875q-0.375 0.296875 -0.78125 0.296875q-0.375 0 -0.640625 -0.234375q-0.25 -0.25 -0.234375 -0.59375l0.03125 -0.34375q-0.515625 0.53125 -1.40625 0.890625q-0.875 0.34375 -1.890625 0.34375zm10.245438 -0.03125q-0.40625 0 -0.65625 -0.296875q-0.234375 -0.3125 -0.15625 -0.71875l1.625 -7.6875153l-1.796875 0q-0.375 0 -0.609375 -0.234375q-0.21875 -0.234375 -0.21875 -0.578125q0 -0.453125 0.40625 -0.796875q0.40625 -0.34375 0.921875 -0.34375l1.75 0l0.484375 -2.3125q0.078125 -0.40625 0.453125 -0.71875q0.375 -0.328125 0.78125 -0.328125q0.390625 0 0.609375 0.25q0.21875 0.234375 0.21875 0.578125q0 0.15625 -0.015625 0.21875l-0.484375 2.3125l1.8125 0q0.359375 0 0.578125 0.21875q0.21875 0.21875 0.21875 0.5625q0 0.46875 -0.390625 0.828125q-0.390625 0.34375 -0.859375 0.34375l-1.8125 0l-1.625 7.6875153q-0.078125 0.4375 -0.453125 0.734375q-0.375 0.28125 -0.78125 0.28125zm8.73027 -11.562515q-0.34375 0 -0.578125 -0.21875q-0.234375 -0.234375 -0.234375 -0.578125q0 -0.078125 0.03125 -0.25l0.203125 -0.796875q0.078125 -0.40625 0.453125 -0.703125q0.390625 -0.296875 0.78125 -0.296875q0.34375 0 0.578125 0.25q0.234375 0.25 0.234375 0.578125q0 0.140625 -0.03125 0.203125l-0.203125 0.8125q-0.109375 0.421875 -0.46875 0.71875q-0.34375 0.28125 -0.765625 0.28125zm-2.546875 11.531265q-0.359375 0 -0.609375 -0.234375q-0.25 -0.234375 -0.25 -0.59375q0 -0.125 0.03125 -0.21875l1.890625 -8.48439q0.09375 -0.421875 0.46875 -0.734375q0.390625 -0.3125 0.8125 -0.3125q0.390625 0 0.609375 0.234375q0.21875 0.21875 0.21875 0.578125q0 0.15625 -0.015625 0.234375l-1.890625 8.48439q-0.09375 0.4375 -0.46875 0.75q-0.375 0.296875 -0.796875 0.296875zm7.8546295 0q-1.578125 0 -2.53125 -1.046875q-0.953125 -1.0625 -0.953125 -2.71875q0 -1.625 0.921875 -3.2343903q0.921875 -1.609375 2.390625 -2.625q1.46875 -1.03125 3.03125 -1.03125q1.046875 0 1.828125 0.5q0.796875 0.5 1.21875 1.359375q0.421875 0.859375 0.421875 1.9375q0 1.6093903 -0.921875 3.2187653q-0.90625 1.59375 -2.390625 2.625q-1.46875 1.015625 -3.015625 1.015625zm0.375 -1.890625q0.921875 0 1.796875 -0.6875q0.875 -0.703125 1.421875 -1.765625q0.5625 -1.078125 0.5625 -2.09375q0 -1.0000153 -0.5 -1.6562653q-0.484375 -0.671875 -1.296875 -0.671875q-0.90625 0 -1.78125 0.734375q-0.875 0.71875 -1.421875 1.8281403q-0.546875 1.09375 -0.546875 2.125q0 0.96875 0.46875 1.578125q0.46875 0.609375 1.296875 0.609375zm7.942856 1.890625q-0.40625 0 -0.65625 -0.296875q-0.25 -0.296875 -0.15625 -0.6875l1.828125 -8.656265q0.09375 -0.421875 0.46875 -0.71875q0.390625 -0.296875 0.8125 -0.296875q0.34375 0 0.5625 0.21875q0.234375 0.21875 0.234375 0.5625q0 0.15625 -0.03125 0.234375l-0.34375 1.625q0.875 -1.28125 1.9375 -1.953125q1.0625 -0.671875 2.140625 -0.671875q1.234375 0 1.8125 0.609375q0.59375 0.609375 0.59375 2.03125q0 1.328125 -0.421875 3.2500153l-0.8125 3.734375q-0.078125 0.40625 -0.421875 0.71875q-0.34375 0.296875 -0.765625 0.296875q-0.40625 0 -0.65625 -0.296875q-0.25 -0.3125 -0.15625 -0.703125l0.75 -3.546875q0.40625 -1.890625 0.40625 -2.8281403q0 -0.734375 -0.234375 -1.0q-0.234375 -0.28125 -0.703125 -0.28125q-0.640625 0 -1.40625 0.546875q-0.75 0.546875 -1.515625 1.5468903q-0.671875 0.875 -1.046875 1.703125q-0.375 0.828125 -0.625 2.0l-0.375 1.796875q-0.09375 0.46875 -0.421875 0.765625q-0.328125 0.296875 -0.796875 0.296875z"
fill-rule="nonzero" />
<path fill="#000000"
d="m168.41536 257.4537q0 -3.6718903 1.796875 -7.0781403q1.8125 -3.40625 4.78125 -3.40625q1.921875 0 3.078125 1.296875q1.15625 1.296875 1.15625 4.046875q0 3.625 -1.765625 7.0468903q-1.765625 3.40625 -4.78125 3.40625q-1.921875 0 -3.09375 -1.40625q-1.171875 -1.40625 -1.171875 -3.90625zm3.625 -3.8593903l4.09375 0q0.1875 -1.3125 0.1875 -2.0q0 -2.1875 -1.46875 -2.1875q-1.765625 0 -2.8125 4.1875zm-0.5 2.5468903q-0.1875 1.1875 -0.1875 2.015625q0 0.953125 0.359375 1.5625q0.359375 0.609375 1.078125 0.609375q1.859375 0 2.828125 -4.1875l-4.078125 0z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#6aa84f"
d="m179.14699 80.04855l30.595795 0l0 19.199478l-30.595795 0l0 -19.199478z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#6aa84f"
d="m209.74278 80.04855l30.595795 0l0 19.199478l-30.595795 0l0 -19.199478z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#6aa84f"
d="m240.33858 80.04855l30.59581 0l0 19.199478l-30.59581 0l0 -19.199478z"
fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m179.14699 79.549866l0 20.196854" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m209.74278 79.549866l0 20.196854" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m240.33858 79.549866l0 20.196854" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m270.9344 79.549866l0 20.196854" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m178.6483 80.04855l92.784775 0" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m178.6483 99.24803l92.784775 0" fill-rule="nonzero" />
<path fill="#000000" fill-opacity="0.0"
d="m147.54355 31.879265l154.99213 0l0 55.590553l-154.99213 0z"
fill-rule="evenodd" />
<path fill="#000000"
d="m170.7249 55.32464q-1.484375 0 -2.734375 -0.671875q-1.25 -0.671875 -1.984375 -1.9375q-0.71875 -1.28125 -0.71875 -3.0q0 -1.8125 0.84375 -3.453125q0.84375 -1.65625 2.203125 -2.65625q1.359375 -1.0 2.765625 -1.0q0.59375 0 1.1875 0.1875q0.59375 0.1875 1.0625 0.46875q0.21875 -0.234375 0.546875 -0.234375q0.375 0 0.609375 0.265625q0.25 0.25 0.25 0.59375l0 1.203125q0 0.375 -0.265625 0.65625q-0.265625 0.265625 -0.65625 0.265625q-0.375 0 -0.640625 -0.21875q-0.25 -0.21875 -0.25 -0.515625l0 -0.140625q-0.6875 -0.71875 -1.96875 -0.71875q-0.9375 0 -1.828125 0.75q-0.890625 0.75 -1.453125 1.984375q-0.546875 1.234375 -0.546875 2.5625q0 1.171875 0.484375 2.046875q0.484375 0.859375 1.296875 1.328125q0.8125 0.453125 1.796875 0.453125q1.8125 0 2.984375 -1.5625q0.171875 -0.203125 0.34375 -0.28125q0.171875 -0.078125 0.390625 -0.078125q0.390625 0 0.65625 0.25q0.28125 0.25 0.28125 0.671875q0 0.328125 -0.171875 0.5625q-0.765625 1.078125 -1.9375 1.65625q-1.15625 0.5625 -2.546875 0.5625zm9.571777 0q-1.0625 0 -1.953125 -0.578125q-0.875 -0.578125 -1.390625 -1.578125q-0.5 -1.0 -0.5 -2.171875q0 -1.328125 0.59375 -2.46875q0.609375 -1.140625 1.609375 -1.828125q1.015625 -0.6875 2.171875 -0.6875q1.0625 0 1.9375 0.609375q0.890625 0.59375 1.40625 1.578125q0.515625 0.984375 0.515625 2.125q0 1.34375 -0.59375 2.5q-0.59375 1.140625 -1.625 1.828125q-1.015625 0.671875 -2.171875 0.671875zm0 -1.671875q0.65625 0 1.25 -0.453125q0.609375 -0.453125 0.953125 -1.1875q0.359375 -0.75 0.359375 -1.59375q0 -0.703125 -0.28125 -1.328125q-0.28125 -0.625 -0.765625 -1.0q-0.484375 -0.390625 -1.0625 -0.390625q-0.65625 0 -1.234375 0.453125q-0.5625 0.453125 -0.90625 1.21875q-0.328125 0.75 -0.328125 1.609375q0 0.84375 0.28125 1.4375q0.296875 0.59375 0.75 0.921875q0.46875 0.3125 0.984375 0.3125zm7.301529 1.671875q-0.359375 0 -0.625 -0.25q-0.265625 -0.265625 -0.265625 -0.609375l0 -7.578125q0 -0.359375 0.265625 -0.625q0.28125 -0.265625 0.65625 -0.265625q0.359375 0 0.609375 0.265625q0.25 0.265625 0.25 0.625l0 1.421875q0.53125 -1.109375 1.328125 -1.703125q0.796875 -0.59375 1.75 -0.59375q1.046875 0 1.65625 0.484375q0.625 0.484375 0.90625 1.609375q0.28125 1.109375 0.28125 3.0625l-0.015625 3.265625q0 0.359375 -0.25 0.625q-0.25 0.265625 -0.609375 0.265625q-0.359375 0 -0.640625 -0.265625q-0.265625 -0.265625 -0.265625 -0.609375l0 -3.09375q0 -1.453125 -0.125 -2.234375q-0.125 -0.796875 -0.390625 -1.078125q-0.265625 -0.296875 -0.71875 -0.296875q-0.546875 0 -1.109375 0.484375q-0.5625 0.46875 -1.046875 1.34375q-0.421875 0.765625 -0.59375 1.5q-0.171875 0.71875 -0.171875 1.75l0 1.5625q0 0.421875 -0.234375 0.6875q-0.234375 0.25 -0.640625 0.25zm11.974091 -0.015625q-0.953125 0 -1.71875 -0.484375q-0.75 -0.5 -1.1875 -1.453125q-0.4375 -0.953125 -0.4375 -2.28125q0 -1.609375 0.5 -2.734375q0.515625 -1.140625 1.40625 -1.71875q0.890625 -0.59375 2.078125 -0.59375q0.921875 0 1.984375 0.421875l0.09375 -2.890625q0.03125 -0.375 0.296875 -0.640625q0.265625 -0.28125 0.640625 -0.28125q0.359375 0 0.625 0.28125q0.265625 0.265625 0.25 0.671875l-0.25 10.859375q-0.03125 0.375 -0.296875 0.625q-0.25 0.234375 -0.625 0.234375q-0.359375 0 -0.609375 -0.203125q-0.25 -0.203125 -0.25 -0.546875l0 -0.328125q-0.484375 0.453125 -1.140625 0.765625q-0.65625 0.296875 -1.359375 0.296875zm0.15625 -1.671875q0.625 0 1.171875 -0.265625q0.5625 -0.28125 0.875 -0.671875q0.3125 -0.390625 0.328125 -0.734375l0.078125 -3.671875q-0.328125 -0.21875 -0.859375 -0.390625q-0.515625 -0.1875 -1.0 -0.1875q-1.140625 0 -1.71875 0.859375q-0.578125 0.84375 -0.578125 2.46875q0 1.34375 0.453125 1.96875q0.46875 0.625 1.25 0.625zm7.847946 -8.40625q-0.359375 0 -0.625 -0.28125q-0.265625 -0.28125 -0.25 -0.625l0.015625 -0.6875q0.015625 -0.359375 0.28125 -0.625q0.265625 -0.265625 0.609375 -0.265625q0.359375 0 0.625 0.28125q0.28125 0.28125 0.265625 0.625l-0.03125 0.703125q-0.015625 0.375 -0.265625 0.625q-0.25 0.25 -0.625 0.25zm-0.09375 10.09375q-0.375 0 -0.640625 -0.265625q-0.265625 -0.265625 -0.265625 -0.640625l0.09375 -7.4375q0 -0.359375 0.265625 -0.625q0.265625 -0.28125 0.640625 -0.28125q0.390625 0 0.65625 0.28125q0.265625 0.265625 0.265625 0.625l-0.09375 7.4375q0 0.375 -0.28125 0.640625q-0.265625 0.265625 -0.640625 0.265625zm5.2365875 0.015625q-0.34375 0 -0.625 -0.25q-0.265625 -0.265625 -0.265625 -0.625l0 -6.65625l-1.46875 0q-0.375 0 -0.625 -0.265625q-0.265625 -0.28125 -0.265625 -0.625q0 -0.421875 0.28125 -0.671875q0.265625 -0.25 0.640625 -0.25l1.4375 0l0 -2.0q0 -0.34375 0.265625 -0.625q0.265625 -0.28125 0.625 -0.28125q0.40625 0 0.65625 0.28125q0.25 0.265625 0.25 0.625l0 2.0l1.578125 0q0.375 0 0.640625 0.265625q0.265625 0.265625 0.265625 0.65625q0 0.34375 -0.265625 0.625q-0.265625 0.265625 -0.640625 0.265625l-1.578125 0l0 6.65625q0 0.359375 -0.265625 0.625q-0.265625 0.25 -0.640625 0.25zm5.5098724 -10.109375q-0.359375 0 -0.625 -0.28125q-0.265625 -0.28125 -0.25 -0.625l0.015625 -0.6875q0.015625 -0.359375 0.28125 -0.625q0.265625 -0.265625 0.609375 -0.265625q0.359375 0 0.625 0.28125q0.28125 0.28125 0.265625 0.625l-0.03125 0.703125q-0.015625 0.375 -0.265625 0.625q-0.25 0.25 -0.625 0.25zm-0.09375 10.09375q-0.375 0 -0.640625 -0.265625q-0.265625 -0.265625 -0.265625 -0.640625l0.09375 -7.4375q0 -0.359375 0.265625 -0.625q0.265625 -0.28125 0.640625 -0.28125q0.390625 0 0.65625 0.28125q0.265625 0.265625 0.265625 0.625l-0.09375 7.4375q0 0.375 -0.28125 0.640625q-0.265625 0.265625 -0.640625 0.265625zm6.8615875 0q-1.0625 0 -1.953125 -0.578125q-0.875 -0.578125 -1.390625 -1.578125q-0.5 -1.0 -0.5 -2.171875q0 -1.328125 0.59375 -2.46875q0.609375 -1.140625 1.609375 -1.828125q1.015625 -0.6875 2.171875 -0.6875q1.0625 0 1.9375 0.609375q0.890625 0.59375 1.40625 1.578125q0.515625 0.984375 0.515625 2.125q0 1.34375 -0.59375 2.5q-0.59375 1.140625 -1.625 1.828125q-1.015625 0.671875 -2.171875 0.671875zm0 -1.671875q0.65625 0 1.25 -0.453125q0.609375 -0.453125 0.953125 -1.1875q0.359375 -0.75 0.359375 -1.59375q0 -0.703125 -0.28125 -1.328125q-0.28125 -0.625 -0.765625 -1.0q-0.484375 -0.390625 -1.0625 -0.390625q-0.65625 0 -1.234375 0.453125q-0.5625 0.453125 -0.90625 1.21875q-0.328125 0.75 -0.328125 1.609375q0 0.84375 0.28125 1.4375q0.296875 0.59375 0.75 0.921875q0.46875 0.3125 0.984375 0.3125zm7.301529 1.671875q-0.359375 0 -0.625 -0.25q-0.265625 -0.265625 -0.265625 -0.609375l0 -7.578125q0 -0.359375 0.265625 -0.625q0.28125 -0.265625 0.65625 -0.265625q0.359375 0 0.609375 0.265625q0.25 0.265625 0.25 0.625l0 1.421875q0.53125 -1.109375 1.328125 -1.703125q0.796875 -0.59375 1.75 -0.59375q1.046875 0 1.65625 0.484375q0.625 0.484375 0.90625 1.609375q0.28125 1.109375 0.28125 3.0625l-0.015625 3.265625q0 0.359375 -0.25 0.625q-0.25 0.265625 -0.609375 0.265625q-0.359375 0 -0.640625 -0.265625q-0.265625 -0.265625 -0.265625 -0.609375l0 -3.09375q0 -1.453125 -0.125 -2.234375q-0.125 -0.796875 -0.390625 -1.078125q-0.265625 -0.296875 -0.71875 -0.296875q-0.546875 0 -1.109375 0.484375q-0.5625 0.46875 -1.046875 1.34375q-0.421875 0.765625 -0.59375 1.5q-0.171875 0.71875 -0.171875 1.75l0 1.5625q0 0.421875 -0.234375 0.6875q-0.234375 0.25 -0.640625 0.25zm13.255341 0q-1.3125 0 -2.375 -0.5625q-1.046875 -0.578125 -1.65625 -1.625q-0.59375 -1.0625 -0.59375 -2.5q0 -1.171875 0.53125 -2.234375q0.53125 -1.078125 1.484375 -1.734375q0.96875 -0.65625 2.203125 -0.65625q1.140625 0 1.953125 0.46875q0.8125 0.453125 1.21875 1.09375q0.40625 0.625 0.40625 1.125q0 0.25 -0.15625 0.46875q-0.140625 0.21875 -0.421875 0.390625l-5.0 2.875q0.265625 0.578125 0.921875 0.90625q0.671875 0.3125 1.484375 0.3125q1.09375 0 1.859375 -0.8125q0.296875 -0.296875 0.609375 -0.296875q0.375 0 0.625 0.25q0.25 0.234375 0.25 0.59375q0 0.296875 -0.21875 0.5625q-0.53125 0.578125 -1.40625 0.984375q-0.859375 0.390625 -1.71875 0.390625zm1.328125 -6.84375q-0.21875 -0.390625 -0.640625 -0.578125q-0.40625 -0.203125 -0.953125 -0.203125q-1.0625 0 -1.78125 0.734375q-0.71875 0.71875 -0.84375 1.78125q-0.03125 0.28125 -0.03125 0.671875l4.25 -2.40625zm6.5508423 6.828125q-0.953125 0 -1.71875 -0.484375q-0.75 -0.5 -1.1875 -1.453125q-0.4375 -0.953125 -0.4375 -2.28125q0 -1.609375 0.5 -2.734375q0.515625 -1.140625 1.40625 -1.71875q0.890625 -0.59375 2.078125 -0.59375q0.921875 0 1.9843597 0.421875l0.09375 -2.890625q0.03125 -0.375 0.296875 -0.640625q0.265625 -0.28125 0.640625 -0.28125q0.359375 0 0.625 0.28125q0.265625 0.265625 0.25 0.671875l-0.25 10.859375q-0.03125 0.375 -0.296875 0.625q-0.25 0.234375 -0.625 0.234375q-0.359375 0 -0.609375 -0.203125q-0.24998474 -0.203125 -0.24998474 -0.546875l0 -0.328125q-0.484375 0.453125 -1.140625 0.765625q-0.65625 0.296875 -1.359375 0.296875zm0.15625 -1.671875q0.625 0 1.171875 -0.265625q0.5625 -0.28125 0.875 -0.671875q0.3125 -0.390625 0.328125 -0.734375l0.07810974 -3.671875q-0.32810974 -0.21875 -0.85935974 -0.390625q-0.515625 -0.1875 -1.0 -0.1875q-1.140625 0 -1.71875 0.859375q-0.578125 0.84375 -0.578125 2.46875q0 1.34375 0.453125 1.96875q0.46875 0.625 1.25 0.625zm16.723953 1.6875q-1.0625 0 -1.953125 -0.578125q-0.875 -0.578125 -1.390625 -1.578125q-0.5 -1.0 -0.5 -2.171875q0 -1.328125 0.59375 -2.46875q0.609375 -1.140625 1.609375 -1.828125q1.015625 -0.6875 2.171875 -0.6875q1.0625 0 1.9375 0.609375q0.890625 0.59375 1.40625 1.578125q0.515625 0.984375 0.515625 2.125q0 1.34375 -0.59375 2.5q-0.59375 1.140625 -1.625 1.828125q-1.015625 0.671875 -2.171875 0.671875zm0 -1.671875q0.65625 0 1.25 -0.453125q0.609375 -0.453125 0.953125 -1.1875q0.359375 -0.75 0.359375 -1.59375q0 -0.703125 -0.28125 -1.328125q-0.28125 -0.625 -0.765625 -1.0q-0.484375 -0.390625 -1.0625 -0.390625q-0.65625 0 -1.234375 0.453125q-0.5625 0.453125 -0.90625 1.21875q-0.328125 0.75 -0.328125 1.609375q0 0.84375 0.28125 1.4375q0.296875 0.59375 0.75 0.921875q0.46875 0.3125 0.984375 0.3125zm7.3015137 1.671875q-0.359375 0 -0.625 -0.25q-0.265625 -0.265625 -0.265625 -0.609375l0 -7.578125q0 -0.359375 0.265625 -0.625q0.28125 -0.265625 0.65625 -0.265625q0.359375 0 0.609375 0.265625q0.25 0.265625 0.25 0.625l0 1.421875q0.53125 -1.109375 1.328125 -1.703125q0.796875 -0.59375 1.75 -0.59375q1.046875 0 1.65625 0.484375q0.625 0.484375 0.90625 1.609375q0.28125 1.109375 0.28125 3.0625l-0.015625 3.265625q0 0.359375 -0.25 0.625q-0.25 0.265625 -0.609375 0.265625q-0.359375 0 -0.640625 -0.265625q-0.265625 -0.265625 -0.265625 -0.609375l0 -3.09375q0 -1.453125 -0.125 -2.234375q-0.125 -0.796875 -0.390625 -1.078125q-0.265625 -0.296875 -0.71875 -0.296875q-0.546875 0 -1.109375 0.484375q-0.5625 0.46875 -1.046875 1.34375q-0.421875 0.765625 -0.59375 1.5q-0.171875 0.71875 -0.171875 1.75l0 1.5625q0 0.421875 -0.234375 0.6875q-0.234375 0.25 -0.640625 0.25z"
fill-rule="nonzero" />
<path fill="#000000"
d="m170.81418 73.24464q-0.359375 0 -0.640625 -0.25q-0.265625 -0.265625 -0.265625 -0.609375l0 -7.546875q0 -0.375 0.265625 -0.640625q0.265625 -0.265625 0.640625 -0.265625q0.375 0 0.640625 0.265625q0.28125 0.265625 0.28125 0.640625l0 0.546875q0.90625 -1.484375 2.5 -1.484375q0.9375 0 1.359375 0.53125q0.421875 0.515625 0.421875 1.234375l0 0.5625q0 0.375 -0.25 0.625q-0.25 0.25 -0.625 0.25q-0.453125 0 -0.671875 -0.25q-0.203125 -0.265625 -0.203125 -0.640625l0 -0.234375q0 -0.3125 -0.40625 -0.3125q-0.34375 0 -0.78125 0.34375q-0.4375 0.328125 -0.71875 0.703125q-0.390625 0.484375 -0.515625 0.96875q-0.109375 0.484375 -0.109375 1.15625l-0.03125 3.546875q0 0.34375 -0.28125 0.609375q-0.265625 0.25 -0.609375 0.25zm10.067841 0q-1.3125 0 -2.375 -0.5625q-1.046875 -0.578125 -1.65625 -1.625q-0.59375 -1.0625 -0.59375 -2.5q0 -1.171875 0.53125 -2.234375q0.53125 -1.078125 1.484375 -1.734375q0.96875 -0.65625 2.203125 -0.65625q1.140625 0 1.953125 0.46875q0.8125 0.453125 1.21875 1.09375q0.40625 0.625 0.40625 1.125q0 0.25 -0.15625 0.46875q-0.140625 0.21875 -0.421875 0.390625l-5.0 2.875q0.265625 0.578125 0.921875 0.90625q0.671875 0.3125 1.484375 0.3125q1.09375 0 1.859375 -0.8125q0.296875 -0.296875 0.609375 -0.296875q0.375 0 0.625 0.25q0.25 0.234375 0.25 0.59375q0 0.296875 -0.21875 0.5625q-0.53125 0.578125 -1.40625 0.984375q-0.859375 0.390625 -1.71875 0.390625zm1.328125 -6.84375q-0.21875 -0.390625 -0.640625 -0.578125q-0.40625 -0.203125 -0.953125 -0.203125q-1.0625 0 -1.78125 0.734375q-0.71875 0.71875 -0.84375 1.78125q-0.03125 0.28125 -0.03125 0.671875l4.25 -2.40625zm7.2852173 6.84375q-1.125 0 -2.0625 -0.53125q-0.921875 -0.546875 -1.46875 -1.484375q-0.546875 -0.953125 -0.546875 -2.0625q0 -1.28125 0.59375 -2.484375q0.609375 -1.21875 1.578125 -1.984375q0.96875 -0.765625 2.046875 -0.765625q0.984375 0 1.59375 0.3125q0.609375 0.3125 1.1875 0.984375q0.28125 0.296875 0.28125 0.65625q0 0.34375 -0.265625 0.59375q-0.25 0.25 -0.625 0.25q-0.25 0 -0.390625 -0.0625q-0.140625 -0.078125 -0.328125 -0.296875q-0.3125 -0.359375 -0.625 -0.546875q-0.296875 -0.1875 -0.8125 -0.1875q-0.578125 0 -1.171875 0.5q-0.578125 0.5 -0.9375 1.3125q-0.359375 0.8125 -0.359375 1.671875q0 0.75 0.3125 1.296875q0.328125 0.546875 0.84375 0.84375q0.53125 0.296875 1.109375 0.296875q0.75 0 1.21875 -0.28125q0.484375 -0.28125 0.859375 -0.765625q0.34375 -0.46875 0.796875 -0.46875q0.359375 0 0.609375 0.25q0.265625 0.25 0.265625 0.5625q0 0.375 -0.203125 0.640625q-0.53125 0.71875 -1.46875 1.234375q-0.9375 0.515625 -2.03125 0.515625zm8.534058 0q-1.0625 0 -1.953125 -0.578125q-0.875 -0.578125 -1.390625 -1.578125q-0.5 -1.0 -0.5 -2.171875q0 -1.328125 0.59375 -2.46875q0.609375 -1.140625 1.609375 -1.828125q1.015625 -0.6875 2.171875 -0.6875q1.0625 0 1.9375 0.609375q0.890625 0.59375 1.40625 1.578125q0.515625 0.984375 0.515625 2.125q0 1.34375 -0.59375 2.5q-0.59375 1.140625 -1.625 1.828125q-1.015625 0.671875 -2.171875 0.671875zm0 -1.671875q0.65625 0 1.25 -0.453125q0.609375 -0.453125 0.953125 -1.1875q0.359375 -0.75 0.359375 -1.59375q0 -0.703125 -0.28125 -1.328125q-0.28125 -0.625 -0.765625 -1.0q-0.484375 -0.390625 -1.0625 -0.390625q-0.65625 0 -1.234375 0.453125q-0.5625 0.453125 -0.90625 1.21875q-0.328125 0.75 -0.328125 1.609375q0 0.84375 0.28125 1.4375q0.296875 0.59375 0.75 0.921875q0.46875 0.3125 0.984375 0.3125zm7.317154 1.671875q-0.359375 0 -0.640625 -0.25q-0.265625 -0.265625 -0.265625 -0.609375l0 -7.546875q0 -0.375 0.265625 -0.640625q0.265625 -0.265625 0.640625 -0.265625q0.375 0 0.640625 0.265625q0.28125 0.265625 0.28125 0.640625l0 0.546875q0.90625 -1.484375 2.5 -1.484375q0.9375 0 1.359375 0.53125q0.421875 0.515625 0.421875 1.234375l0 0.5625q0 0.375 -0.25 0.625q-0.25 0.25 -0.625 0.25q-0.453125 0 -0.671875 -0.25q-0.203125 -0.265625 -0.203125 -0.640625l0 -0.234375q0 -0.3125 -0.40625 -0.3125q-0.34375 0 -0.78125 0.34375q-0.4375 0.328125 -0.71875 0.703125q-0.390625 0.484375 -0.515625 0.96875q-0.109375 0.484375 -0.109375 1.15625l-0.03125 3.546875q0 0.34375 -0.28125 0.609375q-0.265625 0.25 -0.609375 0.25zm8.786591 -0.015625q-0.953125 0 -1.71875 -0.484375q-0.75 -0.5 -1.1875 -1.453125q-0.4375 -0.953125 -0.4375 -2.28125q0 -1.609375 0.5 -2.734375q0.515625 -1.140625 1.40625 -1.71875q0.890625 -0.59375 2.078125 -0.59375q0.921875 0 1.984375 0.421875l0.09375 -2.890625q0.03125 -0.375 0.296875 -0.640625q0.265625 -0.28125 0.640625 -0.28125q0.359375 0 0.625 0.28125q0.265625 0.265625 0.25 0.671875l-0.25 10.859375q-0.03125 0.375 -0.296875 0.625q-0.25 0.234375 -0.625 0.234375q-0.359375 0 -0.609375 -0.203125q-0.25 -0.203125 -0.25 -0.546875l0 -0.328125q-0.484375 0.453125 -1.140625 0.765625q-0.65625 0.296875 -1.359375 0.296875zm0.15625 -1.671875q0.625 0 1.171875 -0.265625q0.5625 -0.28125 0.875 -0.671875q0.3125 -0.390625 0.328125 -0.734375l0.078125 -3.671875q-0.328125 -0.21875 -0.859375 -0.390625q-0.515625 -0.1875 -1.0 -0.1875q-1.140625 0 -1.71875 0.859375q-0.578125 0.84375 -0.578125 2.46875q0 1.34375 0.453125 1.96875q0.46875 0.625 1.25 0.625z"
fill-rule="nonzero" />
<path fill="#000000"
d="m229.91962 63.150887q-0.296875 0 -0.5 -0.1875q-0.203125 -0.203125 -0.203125 -0.5q0 -0.078125 0.03125 -0.21875l0.171875 -0.6875q0.0625 -0.359375 0.390625 -0.625q0.34375 -0.265625 0.6875 -0.265625q0.296875 0 0.5 0.21875q0.203125 0.21875 0.203125 0.515625q0 0.109375 -0.015625 0.171875l-0.1875 0.703125q-0.09375 0.375 -0.40625 0.625q-0.296875 0.25 -0.671875 0.25zm-2.21875 10.09375q-0.3125 0 -0.53125 -0.203125q-0.21875 -0.203125 -0.21875 -0.515625q0 -0.125 0.015625 -0.1875l1.671875 -7.4375q0.0625 -0.359375 0.390625 -0.625q0.34375 -0.28125 0.71875 -0.28125q0.34375 0 0.53125 0.203125q0.203125 0.1875 0.203125 0.5q0 0.140625 -0.015625 0.203125l-1.671875 7.4375q-0.078125 0.375 -0.40625 0.640625q-0.3125 0.265625 -0.6875 0.265625z"
fill-rule="nonzero" />
<path fill="#000000"
d="m239.81346 73.22901q-0.375 0 -0.625 -0.234375q-0.234375 -0.25 -0.234375 -0.609375l0 -10.828125q0 -0.375 0.25 -0.609375q0.265625 -0.234375 0.640625 -0.234375q0.359375 0 0.609375 0.234375q0.265625 0.234375 0.265625 0.609375l0 3.65625q0.546875 -0.609375 1.1875 -0.859375q0.65625 -0.265625 1.4375 -0.265625q0.9375 0 1.703125 0.484375q0.765625 0.46875 1.21875 1.40625q0.453125 0.921875 0.453125 2.25q0 2.390625 -1.09375 3.65625q-1.09375 1.25 -3.0 1.25q-0.53125 0 -1.125 -0.171875q-0.578125 -0.1875 -0.890625 -0.40625l0 0.109375q0 0.1875 -0.234375 0.375q-0.234375 0.1875 -0.5625 0.1875zm2.671875 -1.84375q1.3125 0 1.890625 -0.8125q0.59375 -0.828125 0.59375 -2.34375q0 -1.15625 -0.453125 -1.75q-0.4375 -0.59375 -1.203125 -0.59375q-0.703125 0 -1.3125 0.265625q-0.59375 0.25 -0.9375 0.703125q-0.34375 0.4375 -0.34375 0.984375l0 3.21875q0.953125 0.328125 1.765625 0.328125zm10.278915 1.859375q-1.3125 0 -2.375 -0.5625q-1.046875 -0.578125 -1.65625 -1.625q-0.59375 -1.0625 -0.59375 -2.5q0 -1.171875 0.53125 -2.234375q0.53125 -1.078125 1.484375 -1.734375q0.96875 -0.65625 2.203125 -0.65625q1.140625 0 1.953125 0.46875q0.8125 0.453125 1.21875 1.09375q0.40625 0.625 0.40625 1.125q0 0.25 -0.15625 0.46875q-0.140625 0.21875 -0.421875 0.390625l-5.0 2.875q0.265625 0.578125 0.921875 0.90625q0.671875 0.3125 1.484375 0.3125q1.09375 0 1.859375 -0.8125q0.296875 -0.296875 0.609375 -0.296875q0.375 0 0.625 0.25q0.25 0.234375 0.25 0.59375q0 0.296875 -0.21875 0.5625q-0.53125 0.578125 -1.40625 0.984375q-0.859375 0.390625 -1.71875 0.390625zm1.328125 -6.84375q-0.21875 -0.390625 -0.640625 -0.578125q-0.40625 -0.203125 -0.953125 -0.203125q-1.0625 0 -1.78125 0.734375q-0.71875 0.71875 -0.84375 1.78125q-0.03125 0.28125 -0.03125 0.671875l4.25 -2.40625zm4.8164673 -3.25q-0.359375 0 -0.625 -0.28125q-0.265625 -0.28125 -0.25 -0.625l0.015625 -0.6875q0.015625 -0.359375 0.28125 -0.625q0.265625 -0.265625 0.609375 -0.265625q0.359375 0 0.625 0.28125q0.28125 0.28125 0.265625 0.625l-0.03125 0.703125q-0.015625 0.375 -0.265625 0.625q-0.25 0.25 -0.625 0.25zm-0.09375 10.09375q-0.375 0 -0.640625 -0.265625q-0.265625 -0.265625 -0.265625 -0.640625l0.09375 -7.4375q0 -0.359375 0.265625 -0.625q0.265625 -0.28125 0.640625 -0.28125q0.390625 0 0.65625 0.28125q0.265625 0.265625 0.265625 0.625l-0.09375 7.4375q0 0.375 -0.28125 0.640625q-0.265625 0.265625 -0.640625 0.265625zm4.517853 0q-0.359375 0 -0.625 -0.25q-0.265625 -0.265625 -0.265625 -0.609375l0 -7.578125q0 -0.359375 0.265625 -0.625q0.28125 -0.265625 0.65625 -0.265625q0.359375 0 0.609375 0.265625q0.25 0.265625 0.25 0.625l0 1.421875q0.53125 -1.109375 1.328125 -1.703125q0.796875 -0.59375 1.75 -0.59375q1.046875 0 1.65625 0.484375q0.625 0.484375 0.90625 1.609375q0.28125 1.109375 0.28125 3.0625l-0.015625 3.265625q0 0.359375 -0.25 0.625q-0.25 0.265625 -0.609375 0.265625q-0.359375 0 -0.640625 -0.265625q-0.265625 -0.265625 -0.265625 -0.609375l0 -3.09375q0 -1.453125 -0.125 -2.234375q-0.125 -0.796875 -0.390625 -1.078125q-0.265625 -0.296875 -0.71875 -0.296875q-0.546875 0 -1.109375 0.484375q-0.5625 0.46875 -1.046875 1.34375q-0.421875 0.765625 -0.59375 1.5q-0.171875 0.71875 -0.171875 1.75l0 1.5625q0 0.421875 -0.234375 0.6875q-0.234375 0.25 -0.640625 0.25zm12.427216 3.140625q-1.078125 0 -2.203125 -0.265625q-0.296875 -0.078125 -0.5 -0.328125q-0.1875 -0.234375 -0.1875 -0.546875q0 -0.375 0.265625 -0.640625q0.265625 -0.25 0.625 -0.25q0.21875 0 0.59375 0.09375q0.078125 0.015625 0.46875 0.078125q0.390625 0.0625 0.921875 0.0625q1.921875 0 2.0625 -1.921875l0.03125 -0.453125q-0.3125 0.4375 -0.84375 0.65625q-0.515625 0.203125 -1.109375 0.203125q-0.984375 0 -1.859375 -0.484375q-0.859375 -0.5 -1.390625 -1.4375q-0.515625 -0.9375 -0.515625 -2.21875q0 -1.3125 0.578125 -2.46875q0.59375 -1.15625 1.609375 -1.84375q1.015625 -0.703125 2.25 -0.703125q0.6875 0 1.546875 0.34375q0.15625 -0.53125 0.859375 -0.53125q0.390625 0 0.640625 0.21875q0.25 0.21875 0.234375 0.703125l-0.1875 7.875q-0.03125 1.921875 -1.09375 2.890625q-1.0625 0.96875 -2.796875 0.96875zm0.015625 -5.0625q1.03125 0 1.609375 -0.734375q0.578125 -0.734375 0.578125 -2.28125l0 -2.265625q-0.265625 -0.140625 -0.640625 -0.234375q-0.359375 -0.09375 -0.6875 -0.09375q-0.765625 0 -1.40625 0.40625q-0.640625 0.390625 -1.03125 1.109375q-0.375 0.703125 -0.375 1.578125q0 1.140625 0.5625 1.828125q0.5625 0.6875 1.390625 0.6875z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#c27ba0"
d="m179.14699 399.0315l30.595795 0l0 19.199463l-30.595795 0l0 -19.199463z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#c27ba0"
d="m209.74278 399.0315l30.595795 0l0 19.199463l-30.595795 0l0 -19.199463z"
fill-rule="nonzero" />
<path shape-rendering="crispEdges" fill="#c27ba0"
d="m240.33858 399.0315l30.59581 0l0 19.199463l-30.59581 0l0 -19.199463z"
fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m179.14699 398.5328l0 20.196838" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m209.74278 398.5328l0 20.196838" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m240.33858 398.5328l0 20.196838" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m270.9344 398.5328l0 20.196838" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m178.6483 399.0315l92.784775 0" fill-rule="nonzero" />
<path stroke="#434343" stroke-width="1.0" stroke-linecap="butt"
d="m178.6483 418.23096l92.784775 0" fill-rule="nonzero" />
<path fill="#000000" fill-opacity="0.0"
d="m147.54355 350.8632l154.99213 0l0 55.590546l-154.99213 0z"
fill-rule="evenodd" />
<path fill="#000000"
d="m170.7249 374.30856q-1.484375 0 -2.734375 -0.671875q-1.25 -0.671875 -1.984375 -1.9375q-0.71875 -1.28125 -0.71875 -3.0q0 -1.8125 0.84375 -3.453125q0.84375 -1.65625 2.203125 -2.65625q1.359375 -1.0 2.765625 -1.0q0.59375 0 1.1875 0.1875q0.59375 0.1875 1.0625 0.46875q0.21875 -0.234375 0.546875 -0.234375q0.375 0 0.609375 0.265625q0.25 0.25 0.25 0.59375l0 1.203125q0 0.375 -0.265625 0.65625q-0.265625 0.265625 -0.65625 0.265625q-0.375 0 -0.640625 -0.21875q-0.25 -0.21875 -0.25 -0.515625l0 -0.140625q-0.6875 -0.71875 -1.96875 -0.71875q-0.9375 0 -1.828125 0.75q-0.890625 0.75 -1.453125 1.984375q-0.546875 1.234375 -0.546875 2.5625q0 1.171875 0.484375 2.046875q0.484375 0.859375 1.296875 1.328125q0.8125 0.453125 1.796875 0.453125q1.8125 0 2.984375 -1.5625q0.171875 -0.203125 0.34375 -0.28125q0.171875 -0.078125 0.390625 -0.078125q0.390625 0 0.65625 0.25q0.28125 0.25 0.28125 0.671875q0 0.328125 -0.171875 0.5625q-0.765625 1.078125 -1.9375 1.65625q-1.15625 0.5625 -2.546875 0.5625zm9.571777 0q-1.0625 0 -1.953125 -0.578125q-0.875 -0.578125 -1.390625 -1.578125q-0.5 -1.0 -0.5 -2.171875q0 -1.328125 0.59375 -2.46875q0.609375 -1.140625 1.609375 -1.828125q1.015625 -0.6875 2.171875 -0.6875q1.0625 0 1.9375 0.609375q0.890625 0.59375 1.40625 1.578125q0.515625 0.984375 0.515625 2.125q0 1.34375 -0.59375 2.5q-0.59375 1.140625 -1.625 1.828125q-1.015625 0.671875 -2.171875 0.671875zm0 -1.671875q0.65625 0 1.25 -0.453125q0.609375 -0.453125 0.953125 -1.1875q0.359375 -0.75 0.359375 -1.59375q0 -0.703125 -0.28125 -1.328125q-0.28125 -0.625 -0.765625 -1.0q-0.484375 -0.390625 -1.0625 -0.390625q-0.65625 0 -1.234375 0.453125q-0.5625 0.453125 -0.90625 1.21875q-0.328125 0.75 -0.328125 1.609375q0 0.84375 0.28125 1.4375q0.296875 0.59375 0.75 0.921875q0.46875 0.3125 0.984375 0.3125zm7.301529 1.671875q-0.359375 0 -0.625 -0.25q-0.265625 -0.265625 -0.265625 -0.609375l0 -7.578125q0 -0.359375 0.265625 -0.625q0.28125 -0.265625 0.65625 -0.265625q0.359375 0 0.609375 0.265625q0.25 0.265625 0.25 0.625l0 1.421875q0.53125 -1.109375 1.328125 -1.703125q0.796875 -0.59375 1.75 -0.59375q1.046875 0 1.65625 0.484375q0.625 0.484375 0.90625 1.609375q0.28125 1.109375 0.28125 3.0625l-0.015625 3.265625q0 0.359375 -0.25 0.625q-0.25 0.265625 -0.609375 0.265625q-0.359375 0 -0.640625 -0.265625q-0.265625 -0.265625 -0.265625 -0.609375l0 -3.09375q0 -1.453125 -0.125 -2.234375q-0.125 -0.796875 -0.390625 -1.078125q-0.265625 -0.296875 -0.71875 -0.296875q-0.546875 0 -1.109375 0.484375q-0.5625 0.46875 -1.046875 1.34375q-0.421875 0.765625 -0.59375 1.5q-0.171875 0.71875 -0.171875 1.75l0 1.5625q0 0.421875 -0.234375 0.6875q-0.234375 0.25 -0.640625 0.25zm11.974091 -0.015625q-0.953125 0 -1.71875 -0.484375q-0.75 -0.5 -1.1875 -1.453125q-0.4375 -0.953125 -0.4375 -2.28125q0 -1.609375 0.5 -2.734375q0.515625 -1.140625 1.40625 -1.71875q0.890625 -0.59375 2.078125 -0.59375q0.921875 0 1.984375 0.421875l0.09375 -2.890625q0.03125 -0.375 0.296875 -0.640625q0.265625 -0.28125 0.640625 -0.28125q0.359375 0 0.625 0.28125q0.265625 0.265625 0.25 0.671875l-0.25 10.859375q-0.03125 0.375 -0.296875 0.625q-0.25 0.234375 -0.625 0.234375q-0.359375 0 -0.609375 -0.203125q-0.25 -0.203125 -0.25 -0.546875l0 -0.328125q-0.484375 0.453125 -1.140625 0.765625q-0.65625 0.296875 -1.359375 0.296875zm0.15625 -1.671875q0.625 0 1.171875 -0.265625q0.5625 -0.28125 0.875 -0.671875q0.3125 -0.390625 0.328125 -0.734375l0.078125 -3.671875q-0.328125 -0.21875 -0.859375 -0.390625q-0.515625 -0.1875 -1.0 -0.1875q-1.140625 0 -1.71875 0.859375q-0.578125 0.84375 -0.578125 2.46875q0 1.34375 0.453125 1.96875q0.46875 0.625 1.25 0.625zm7.847946 -8.40625q-0.359375 0 -0.625 -0.28125q-0.265625 -0.28125 -0.25 -0.625l0.015625 -0.6875q0.015625 -0.359375 0.28125 -0.625q0.265625 -0.265625 0.609375 -0.265625q0.359375 0 0.625 0.28125q0.28125 0.28125 0.265625 0.625l-0.03125 0.703125q-0.015625 0.375 -0.265625 0.625q-0.25 0.25 -0.625 0.25zm-0.09375 10.09375q-0.375 0 -0.640625 -0.265625q-0.265625 -0.265625 -0.265625 -0.640625l0.09375 -7.4375q0 -0.359375 0.265625 -0.625q0.265625 -0.28125 0.640625 -0.28125q0.390625 0 0.65625 0.28125q0.265625 0.265625 0.265625 0.625l-0.09375 7.4375q0 0.375 -0.28125 0.640625q-0.265625 0.265625 -0.640625 0.265625zm5.2365875 0.015625q-0.34375 0 -0.625 -0.25q-0.265625 -0.265625 -0.265625 -0.625l0 -6.65625l-1.46875 0q-0.375 0 -0.625 -0.265625q-0.265625 -0.28125 -0.265625 -0.625q0 -0.421875 0.28125 -0.671875q0.265625 -0.25 0.640625 -0.25l1.4375 0l0 -2.0q0 -0.34375 0.265625 -0.625q0.265625 -0.28125 0.625 -0.28125q0.40625 0 0.65625 0.28125q0.25 0.265625 0.25 0.625l0 2.0l1.578125 0q0.375 0 0.640625 0.265625q0.265625 0.265625 0.265625 0.65625q0 0.34375 -0.265625 0.625q-0.265625 0.265625 -0.640625 0.265625l-1.578125 0l0 6.65625q0 0.359375 -0.265625 0.625q-0.265625 0.25 -0.640625 0.25zm5.5098724 -10.109375q-0.359375 0 -0.625 -0.28125q-0.265625 -0.28125 -0.25 -0.625l0.015625 -0.6875q0.015625 -0.359375 0.28125 -0.625q0.265625 -0.265625 0.609375 -0.265625q0.359375 0 0.625 0.28125q0.28125 0.28125 0.265625 0.625l-0.03125 0.703125q-0.015625 0.375 -0.265625 0.625q-0.25 0.25 -0.625 0.25zm-0.09375 10.09375q-0.375 0 -0.640625 -0.265625q-0.265625 -0.265625 -0.265625 -0.640625l0.09375 -7.4375q0 -0.359375 0.265625 -0.625q0.265625 -0.28125 0.640625 -0.28125q0.390625 0 0.65625 0.28125q0.265625 0.265625 0.265625 0.625l-0.09375 7.4375q0 0.375 -0.28125 0.640625q-0.265625 0.265625 -0.640625 0.265625zm6.8615875 0q-1.0625 0 -1.953125 -0.578125q-0.875 -0.578125 -1.390625 -1.578125q-0.5 -1.0 -0.5 -2.171875q0 -1.328125 0.59375 -2.46875q0.609375 -1.140625 1.609375 -1.828125q1.015625 -0.6875 2.171875 -0.6875q1.0625 0 1.9375 0.609375q0.890625 0.59375 1.40625 1.578125q0.515625 0.984375 0.515625 2.125q0 1.34375 -0.59375 2.5q-0.59375 1.140625 -1.625 1.828125q-1.015625 0.671875 -2.171875 0.671875zm0 -1.671875q0.65625 0 1.25 -0.453125q0.609375 -0.453125 0.953125 -1.1875q0.359375 -0.75 0.359375 -1.59375q0 -0.703125 -0.28125 -1.328125q-0.28125 -0.625 -0.765625 -1.0q-0.484375 -0.390625 -1.0625 -0.390625q-0.65625 0 -1.234375 0.453125q-0.5625 0.453125 -0.90625 1.21875q-0.328125 0.75 -0.328125 1.609375q0 0.84375 0.28125 1.4375q0.296875 0.59375 0.75 0.921875q0.46875 0.3125 0.984375 0.3125zm7.301529 1.671875q-0.359375 0 -0.625 -0.25q-0.265625 -0.265625 -0.265625 -0.609375l0 -7.578125q0 -0.359375 0.265625 -0.625q0.28125 -0.265625 0.65625 -0.265625q0.359375 0 0.609375 0.265625q0.25 0.265625 0.25 0.625l0 1.421875q0.53125 -1.109375 1.328125 -1.703125q0.796875 -0.59375 1.75 -0.59375q1.046875 0 1.65625 0.484375q0.625 0.484375 0.90625 1.609375q0.28125 1.109375 0.28125 3.0625l-0.015625 3.265625q0 0.359375 -0.25 0.625q-0.25 0.265625 -0.609375 0.265625q-0.359375 0 -0.640625 -0.265625q-0.265625 -0.265625 -0.265625 -0.609375l0 -3.09375q0 -1.453125 -0.125 -2.234375q-0.125 -0.796875 -0.390625 -1.078125q-0.265625 -0.296875 -0.71875 -0.296875q-0.546875 0 -1.109375 0.484375q-0.5625 0.46875 -1.046875 1.34375q-0.421875 0.765625 -0.59375 1.5q-0.171875 0.71875 -0.171875 1.75l0 1.5625q0 0.421875 -0.234375 0.6875q-0.234375 0.25 -0.640625 0.25zm13.255341 0q-1.3125 0 -2.375 -0.5625q-1.046875 -0.578125 -1.65625 -1.625q-0.59375 -1.0625 -0.59375 -2.5q0 -1.171875 0.53125 -2.234375q0.53125 -1.078125 1.484375 -1.734375q0.96875 -0.65625 2.203125 -0.65625q1.140625 0 1.953125 0.46875q0.8125 0.453125 1.21875 1.09375q0.40625 0.625 0.40625 1.125q0 0.25 -0.15625 0.46875q-0.140625 0.21875 -0.421875 0.390625l-5.0 2.875q0.265625 0.578125 0.921875 0.90625q0.671875 0.3125 1.484375 0.3125q1.09375 0 1.859375 -0.8125q0.296875 -0.296875 0.609375 -0.296875q0.375 0 0.625 0.25q0.25 0.234375 0.25 0.59375q0 0.296875 -0.21875 0.5625q-0.53125 0.578125 -1.40625 0.984375q-0.859375 0.390625 -1.71875 0.390625zm1.328125 -6.84375q-0.21875 -0.390625 -0.640625 -0.578125q-0.40625 -0.203125 -0.953125 -0.203125q-1.0625 0 -1.78125 0.734375q-0.71875 0.71875 -0.84375 1.78125q-0.03125 0.28125 -0.03125 0.671875l4.25 -2.40625zm6.5508423 6.828125q-0.953125 0 -1.71875 -0.484375q-0.75 -0.5 -1.1875 -1.453125q-0.4375 -0.953125 -0.4375 -2.28125q0 -1.609375 0.5 -2.734375q0.515625 -1.140625 1.40625 -1.71875q0.890625 -0.59375 2.078125 -0.59375q0.921875 0 1.9843597 0.421875l0.09375 -2.890625q0.03125 -0.375 0.296875 -0.640625q0.265625 -0.28125 0.640625 -0.28125q0.359375 0 0.625 0.28125q0.265625 0.265625 0.25 0.671875l-0.25 10.859375q-0.03125 0.375 -0.296875 0.625q-0.25 0.234375 -0.625 0.234375q-0.359375 0 -0.609375 -0.203125q-0.24998474 -0.203125 -0.24998474 -0.546875l0 -0.328125q-0.484375 0.453125 -1.140625 0.765625q-0.65625 0.296875 -1.359375 0.296875zm0.15625 -1.671875q0.625 0 1.171875 -0.265625q0.5625 -0.28125 0.875 -0.671875q0.3125 -0.390625 0.328125 -0.734375l0.07810974 -3.671875q-0.32810974 -0.21875 -0.85935974 -0.390625q-0.515625 -0.1875 -1.0 -0.1875q-1.140625 0 -1.71875 0.859375q-0.578125 0.84375 -0.578125 2.46875q0 1.34375 0.453125 1.96875q0.46875 0.625 1.25 0.625zm16.723953 1.6875q-1.0625 0 -1.953125 -0.578125q-0.875 -0.578125 -1.390625 -1.578125q-0.5 -1.0 -0.5 -2.171875q0 -1.328125 0.59375 -2.46875q0.609375 -1.140625 1.609375 -1.828125q1.015625 -0.6875 2.171875 -0.6875q1.0625 0 1.9375 0.609375q0.890625 0.59375 1.40625 1.578125q0.515625 0.984375 0.515625 2.125q0 1.34375 -0.59375 2.5q-0.59375 1.140625 -1.625 1.828125q-1.015625 0.671875 -2.171875 0.671875zm0 -1.671875q0.65625 0 1.25 -0.453125q0.609375 -0.453125 0.953125 -1.1875q0.359375 -0.75 0.359375 -1.59375q0 -0.703125 -0.28125 -1.328125q-0.28125 -0.625 -0.765625 -1.0q-0.484375 -0.390625 -1.0625 -0.390625q-0.65625 0 -1.234375 0.453125q-0.5625 0.453125 -0.90625 1.21875q-0.328125 0.75 -0.328125 1.609375q0 0.84375 0.28125 1.4375q0.296875 0.59375 0.75 0.921875q0.46875 0.3125 0.984375 0.3125zm7.3015137 1.671875q-0.359375 0 -0.625 -0.25q-0.265625 -0.265625 -0.265625 -0.609375l0 -7.578125q0 -0.359375 0.265625 -0.625q0.28125 -0.265625 0.65625 -0.265625q0.359375 0 0.609375 0.265625q0.25 0.265625 0.25 0.625l0 1.421875q0.53125 -1.109375 1.328125 -1.703125q0.796875 -0.59375 1.75 -0.59375q1.046875 0 1.65625 0.484375q0.625 0.484375 0.90625 1.609375q0.28125 1.109375 0.28125 3.0625l-0.015625 3.265625q0 0.359375 -0.25 0.625q-0.25 0.265625 -0.609375 0.265625q-0.359375 0 -0.640625 -0.265625q-0.265625 -0.265625 -0.265625 -0.609375l0 -3.09375q0 -1.453125 -0.125 -2.234375q-0.125 -0.796875 -0.390625 -1.078125q-0.265625 -0.296875 -0.71875 -0.296875q-0.546875 0 -1.109375 0.484375q-0.5625 0.46875 -1.046875 1.34375q-0.421875 0.765625 -0.59375 1.5q-0.171875 0.71875 -0.171875 1.75l0 1.5625q0 0.421875 -0.234375 0.6875q-0.234375 0.25 -0.640625 0.25z"
fill-rule="nonzero" />
<path fill="#000000"
d="m170.81418 392.22855q-0.359375 0 -0.640625 -0.25q-0.265625 -0.265625 -0.265625 -0.609375l0 -7.546875q0 -0.375 0.265625 -0.640625q0.265625 -0.265625 0.640625 -0.265625q0.375 0 0.640625 0.265625q0.28125 0.265625 0.28125 0.640625l0 0.546875q0.90625 -1.484375 2.5 -1.484375q0.9375 0 1.359375 0.53125q0.421875 0.515625 0.421875 1.234375l0 0.5625q0 0.375 -0.25 0.625q-0.25 0.25 -0.625 0.25q-0.453125 0 -0.671875 -0.25q-0.203125 -0.265625 -0.203125 -0.640625l0 -0.234375q0 -0.3125 -0.40625 -0.3125q-0.34375 0 -0.78125 0.34375q-0.4375 0.328125 -0.71875 0.703125q-0.390625 0.484375 -0.515625 0.96875q-0.109375 0.484375 -0.109375 1.15625l-0.03125 3.546875q0 0.34375 -0.28125 0.609375q-0.265625 0.25 -0.609375 0.25zm10.067841 0q-1.3125 0 -2.375 -0.5625q-1.046875 -0.578125 -1.65625 -1.625q-0.59375 -1.0625 -0.59375 -2.5q0 -1.171875 0.53125 -2.234375q0.53125 -1.078125 1.484375 -1.734375q0.96875 -0.65625 2.203125 -0.65625q1.140625 0 1.953125 0.46875q0.8125 0.453125 1.21875 1.09375q0.40625 0.625 0.40625 1.125q0 0.25 -0.15625 0.46875q-0.140625 0.21875 -0.421875 0.390625l-5.0 2.875q0.265625 0.578125 0.921875 0.90625q0.671875 0.3125 1.484375 0.3125q1.09375 0 1.859375 -0.8125q0.296875 -0.296875 0.609375 -0.296875q0.375 0 0.625 0.25q0.25 0.234375 0.25 0.59375q0 0.296875 -0.21875 0.5625q-0.53125 0.578125 -1.40625 0.984375q-0.859375 0.390625 -1.71875 0.390625zm1.328125 -6.84375q-0.21875 -0.390625 -0.640625 -0.578125q-0.40625 -0.203125 -0.953125 -0.203125q-1.0625 0 -1.78125 0.734375q-0.71875 0.71875 -0.84375 1.78125q-0.03125 0.28125 -0.03125 0.671875l4.25 -2.40625zm7.2852173 6.84375q-1.125 0 -2.0625 -0.53125q-0.921875 -0.546875 -1.46875 -1.484375q-0.546875 -0.953125 -0.546875 -2.0625q0 -1.28125 0.59375 -2.484375q0.609375 -1.21875 1.578125 -1.984375q0.96875 -0.765625 2.046875 -0.765625q0.984375 0 1.59375 0.3125q0.609375 0.3125 1.1875 0.984375q0.28125 0.296875 0.28125 0.65625q0 0.34375 -0.265625 0.59375q-0.25 0.25 -0.625 0.25q-0.25 0 -0.390625 -0.0625q-0.140625 -0.078125 -0.328125 -0.296875q-0.3125 -0.359375 -0.625 -0.546875q-0.296875 -0.1875 -0.8125 -0.1875q-0.578125 0 -1.171875 0.5q-0.578125 0.5 -0.9375 1.3125q-0.359375 0.8125 -0.359375 1.671875q0 0.75 0.3125 1.296875q0.328125 0.546875 0.84375 0.84375q0.53125 0.296875 1.109375 0.296875q0.75 0 1.21875 -0.28125q0.484375 -0.28125 0.859375 -0.765625q0.34375 -0.46875 0.796875 -0.46875q0.359375 0 0.609375 0.25q0.265625 0.25 0.265625 0.5625q0 0.375 -0.203125 0.640625q-0.53125 0.71875 -1.46875 1.234375q-0.9375 0.515625 -2.03125 0.515625zm8.534058 0q-1.0625 0 -1.953125 -0.578125q-0.875 -0.578125 -1.390625 -1.578125q-0.5 -1.0 -0.5 -2.171875q0 -1.328125 0.59375 -2.46875q0.609375 -1.140625 1.609375 -1.828125q1.015625 -0.6875 2.171875 -0.6875q1.0625 0 1.9375 0.609375q0.890625 0.59375 1.40625 1.578125q0.515625 0.984375 0.515625 2.125q0 1.34375 -0.59375 2.5q-0.59375 1.140625 -1.625 1.828125q-1.015625 0.671875 -2.171875 0.671875zm0 -1.671875q0.65625 0 1.25 -0.453125q0.609375 -0.453125 0.953125 -1.1875q0.359375 -0.75 0.359375 -1.59375q0 -0.703125 -0.28125 -1.328125q-0.28125 -0.625 -0.765625 -1.0q-0.484375 -0.390625 -1.0625 -0.390625q-0.65625 0 -1.234375 0.453125q-0.5625 0.453125 -0.90625 1.21875q-0.328125 0.75 -0.328125 1.609375q0 0.84375 0.28125 1.4375q0.296875 0.59375 0.75 0.921875q0.46875 0.3125 0.984375 0.3125zm7.317154 1.671875q-0.359375 0 -0.640625 -0.25q-0.265625 -0.265625 -0.265625 -0.609375l0 -7.546875q0 -0.375 0.265625 -0.640625q0.265625 -0.265625 0.640625 -0.265625q0.375 0 0.640625 0.265625q0.28125 0.265625 0.28125 0.640625l0 0.546875q0.90625 -1.484375 2.5 -1.484375q0.9375 0 1.359375 0.53125q0.421875 0.515625 0.421875 1.234375l0 0.5625q0 0.375 -0.25 0.625q-0.25 0.25 -0.625 0.25q-0.453125 0 -0.671875 -0.25q-0.203125 -0.265625 -0.203125 -0.640625l0 -0.234375q0 -0.3125 -0.40625 -0.3125q-0.34375 0 -0.78125 0.34375q-0.4375 0.328125 -0.71875 0.703125q-0.390625 0.484375 -0.515625 0.96875q-0.109375 0.484375 -0.109375 1.15625l-0.03125 3.546875q0 0.34375 -0.28125 0.609375q-0.265625 0.25 -0.609375 0.25zm8.786591 -0.015625q-0.953125 0 -1.71875 -0.484375q-0.75 -0.5 -1.1875 -1.453125q-0.4375 -0.953125 -0.4375 -2.28125q0 -1.609375 0.5 -2.734375q0.515625 -1.140625 1.40625 -1.71875q0.890625 -0.59375 2.078125 -0.59375q0.921875 0 1.984375 0.421875l0.09375 -2.890625q0.03125 -0.375 0.296875 -0.640625q0.265625 -0.28125 0.640625 -0.28125q0.359375 0 0.625 0.28125q0.265625 0.265625 0.25 0.671875l-0.25 10.859375q-0.03125 0.375 -0.296875 0.625q-0.25 0.234375 -0.625 0.234375q-0.359375 0 -0.609375 -0.203125q-0.25 -0.203125 -0.25 -0.546875l0 -0.328125q-0.484375 0.453125 -1.140625 0.765625q-0.65625 0.296875 -1.359375 0.296875zm0.15625 -1.671875q0.625 0 1.171875 -0.265625q0.5625 -0.28125 0.875 -0.671875q0.3125 -0.390625 0.328125 -0.734375l0.078125 -3.671875q-0.328125 -0.21875 -0.859375 -0.390625q-0.515625 -0.1875 -1.0 -0.1875q-1.140625 0 -1.71875 0.859375q-0.578125 0.84375 -0.578125 2.46875q0 1.34375 0.453125 1.96875q0.46875 0.625 1.25 0.625z"
fill-rule="nonzero" />
<path fill="#000000"
d="m229.91962 382.1348q-0.296875 0 -0.5 -0.1875q-0.203125 -0.203125 -0.203125 -0.5q0 -0.078125 0.03125 -0.21875l0.171875 -0.6875q0.0625 -0.359375 0.390625 -0.625q0.34375 -0.265625 0.6875 -0.265625q0.296875 0 0.5 0.21875q0.203125 0.21875 0.203125 0.515625q0 0.109375 -0.015625 0.171875l-0.1875 0.703125q-0.09375 0.375 -0.40625 0.625q-0.296875 0.25 -0.671875 0.25zm-2.21875 10.09375q-0.3125 0 -0.53125 -0.203125q-0.21875 -0.203125 -0.21875 -0.515625q0 -0.125 0.015625 -0.1875l1.671875 -7.4375q0.0625 -0.359375 0.390625 -0.625q0.34375 -0.28125 0.71875 -0.28125q0.34375 0 0.53125 0.203125q0.203125 0.1875 0.203125 0.5q0 0.140625 -0.015625 0.203125l-1.671875 7.4375q-0.078125 0.375 -0.40625 0.640625q-0.3125 0.265625 -0.6875 0.265625z"
fill-rule="nonzero" />
<path fill="#000000"
d="m239.81346 392.21292q-0.375 0 -0.625 -0.234375q-0.234375 -0.25 -0.234375 -0.609375l0 -10.828125q0 -0.375 0.25 -0.609375q0.265625 -0.234375 0.640625 -0.234375q0.359375 0 0.609375 0.234375q0.265625 0.234375 0.265625 0.609375l0 3.65625q0.546875 -0.609375 1.1875 -0.859375q0.65625 -0.265625 1.4375 -0.265625q0.9375 0 1.703125 0.484375q0.765625 0.46875 1.21875 1.40625q0.453125 0.921875 0.453125 2.25q0 2.390625 -1.09375 3.65625q-1.09375 1.25 -3.0 1.25q-0.53125 0 -1.125 -0.171875q-0.578125 -0.1875 -0.890625 -0.40625l0 0.109375q0 0.1875 -0.234375 0.375q-0.234375 0.1875 -0.5625 0.1875zm2.671875 -1.84375q1.3125 0 1.890625 -0.8125q0.59375 -0.828125 0.59375 -2.34375q0 -1.15625 -0.453125 -1.75q-0.4375 -0.59375 -1.203125 -0.59375q-0.703125 0 -1.3125 0.265625q-0.59375 0.25 -0.9375 0.703125q-0.34375 0.4375 -0.34375 0.984375l0 3.21875q0.953125 0.328125 1.765625 0.328125zm10.278915 1.859375q-1.3125 0 -2.375 -0.5625q-1.046875 -0.578125 -1.65625 -1.625q-0.59375 -1.0625 -0.59375 -2.5q0 -1.171875 0.53125 -2.234375q0.53125 -1.078125 1.484375 -1.734375q0.96875 -0.65625 2.203125 -0.65625q1.140625 0 1.953125 0.46875q0.8125 0.453125 1.21875 1.09375q0.40625 0.625 0.40625 1.125q0 0.25 -0.15625 0.46875q-0.140625 0.21875 -0.421875 0.390625l-5.0 2.875q0.265625 0.578125 0.921875 0.90625q0.671875 0.3125 1.484375 0.3125q1.09375 0 1.859375 -0.8125q0.296875 -0.296875 0.609375 -0.296875q0.375 0 0.625 0.25q0.25 0.234375 0.25 0.59375q0 0.296875 -0.21875 0.5625q-0.53125 0.578125 -1.40625 0.984375q-0.859375 0.390625 -1.71875 0.390625zm1.328125 -6.84375q-0.21875 -0.390625 -0.640625 -0.578125q-0.40625 -0.203125 -0.953125 -0.203125q-1.0625 0 -1.78125 0.734375q-0.71875 0.71875 -0.84375 1.78125q-0.03125 0.28125 -0.03125 0.671875l4.25 -2.40625zm4.8164673 -3.25q-0.359375 0 -0.625 -0.28125q-0.265625 -0.28125 -0.25 -0.625l0.015625 -0.6875q0.015625 -0.359375 0.28125 -0.625q0.265625 -0.265625 0.609375 -0.265625q0.359375 0 0.625 0.28125q0.28125 0.28125 0.265625 0.625l-0.03125 0.703125q-0.015625 0.375 -0.265625 0.625q-0.25 0.25 -0.625 0.25zm-0.09375 10.09375q-0.375 0 -0.640625 -0.265625q-0.265625 -0.265625 -0.265625 -0.640625l0.09375 -7.4375q0 -0.359375 0.265625 -0.625q0.265625 -0.28125 0.640625 -0.28125q0.390625 0 0.65625 0.28125q0.265625 0.265625 0.265625 0.625l-0.09375 7.4375q0 0.375 -0.28125 0.640625q-0.265625 0.265625 -0.640625 0.265625zm4.517853 0q-0.359375 0 -0.625 -0.25q-0.265625 -0.265625 -0.265625 -0.609375l0 -7.578125q0 -0.359375 0.265625 -0.625q0.28125 -0.265625 0.65625 -0.265625q0.359375 0 0.609375 0.265625q0.25 0.265625 0.25 0.625l0 1.421875q0.53125 -1.109375 1.328125 -1.703125q0.796875 -0.59375 1.75 -0.59375q1.046875 0 1.65625 0.484375q0.625 0.484375 0.90625 1.609375q0.28125 1.109375 0.28125 3.0625l-0.015625 3.265625q0 0.359375 -0.25 0.625q-0.25 0.265625 -0.609375 0.265625q-0.359375 0 -0.640625 -0.265625q-0.265625 -0.265625 -0.265625 -0.609375l0 -3.09375q0 -1.453125 -0.125 -2.234375q-0.125 -0.796875 -0.390625 -1.078125q-0.265625 -0.296875 -0.71875 -0.296875q-0.546875 0 -1.109375 0.484375q-0.5625 0.46875 -1.046875 1.34375q-0.421875 0.765625 -0.59375 1.5q-0.171875 0.71875 -0.171875 1.75l0 1.5625q0 0.421875 -0.234375 0.6875q-0.234375 0.25 -0.640625 0.25zm12.427216 3.140625q-1.078125 0 -2.203125 -0.265625q-0.296875 -0.078125 -0.5 -0.328125q-0.1875 -0.234375 -0.1875 -0.546875q0 -0.375 0.265625 -0.640625q0.265625 -0.25 0.625 -0.25q0.21875 0 0.59375 0.09375q0.078125 0.015625 0.46875 0.078125q0.390625 0.0625 0.921875 0.0625q1.921875 0 2.0625 -1.921875l0.03125 -0.453125q-0.3125 0.4375 -0.84375 0.65625q-0.515625 0.203125 -1.109375 0.203125q-0.984375 0 -1.859375 -0.484375q-0.859375 -0.5 -1.390625 -1.4375q-0.515625 -0.9375 -0.515625 -2.21875q0 -1.3125 0.578125 -2.46875q0.59375 -1.15625 1.609375 -1.84375q1.015625 -0.703125 2.25 -0.703125q0.6875 0 1.546875 0.34375q0.15625 -0.53125 0.859375 -0.53125q0.390625 0 0.640625 0.21875q0.25 0.21875 0.234375 0.703125l-0.1875 7.875q-0.03125 1.921875 -1.09375 2.890625q-1.0625 0.96875 -2.796875 0.96875zm0.015625 -5.0625q1.03125 0 1.609375 -0.734375q0.578125 -0.734375 0.578125 -2.28125l0 -2.265625q-0.265625 -0.140625 -0.640625 -0.234375q-0.359375 -0.09375 -0.6875 -0.09375q-0.765625 0 -1.40625 0.40625q-0.640625 0.390625 -1.03125 1.109375q-0.375 0.703125 -0.375 1.578125q0 1.140625 0.5625 1.828125q0.5625 0.6875 1.390625 0.6875z"
fill-rule="nonzero" />
<path fill="#7c7ce0"
d="m435.10236 81.62992l114.703186 0l0 -13.535431l16.5094 27.070862l-16.5094 27.07087l0 -13.535431l-114.703186 0z"
fill-rule="evenodd" />
<path fill="#7c7ce0"
d="m435.10236 367.00525l114.703186 0l0 -13.535431l16.5094 27.070862l-16.5094 27.070862l0 -13.535431l-114.703186 0z"
fill-rule="evenodd" />
<path fill="#000000" fill-opacity="0.0"
d="m574.15094 24.150354l154.99207 0l0 142.02911l-154.99207 0z"
fill-rule="evenodd" />
<g transform="matrix(0.5636076115485564 0.0 0.0 0.5636076115485564 574.1509186351706 24.150355118110234)">
<clipPath id="p.3">
<path d="m-1.1368684E-13 0l275.0 0l0 252.0l-275.0 0z"
clip-rule="evenodd" />
</clipPath>
<image clip-path="url(#p.3)" fill="#000" width="275.0"
height="252.0" x="0.0" y="0.0" preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/magic.gif"/>
</g>
<path fill="#000000" fill-opacity="0.0"
d="m574.15094 307.6451l154.99207 0l0 142.02911l-154.99207 0z"
fill-rule="evenodd" />
<g transform="matrix(0.5636076115485564 0.0 0.0 0.5636076115485564 574.1509186351706 307.6451057742782)">
<clipPath id="p.4">
<path d="m-1.1368684E-13 -1.1368684E-13l275.0 0l0 252.0l-275.0 0z"
clip-rule="evenodd" />
</clipPath>
<image clip-path="url(#p.4)" fill="#000" width="275.0"
height="252.0" x="0.0" y="0.0" preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/magic.gif"/>
</g>
</g>
</svg>
</center></p>
<p>This <span class="math">\(\theta\)</span> captures both what the attacker know and doesn't know. The more
uncertainty the attacker has, the "more random" this probability distribution
is. How do we choose <span class="math">\(\theta\)</span>? Great question. We'll come back to it a little
later.</p>
<p>Now: what did we find out?</p>
<h2 id="some-positive-results">Some positive results</h2>
<p>Let's take the very simple case of a referendum, where everybody answers a
question by <em>Yes</em> or <em>No</em>. The attacker is uncertain about whether some people
<span class="math">\(i\)</span> vote <em>Yes</em> or <em>No</em>. This is captured by having each vote be random: <em>Yes</em>
with probability <span class="math">\(p_i\)</span> and <em>No</em> with probability <span class="math">\(1-p_i\)</span>. And we capture "how
uncertain" the attacker is by having a certain number of <span class="math">\(p_i\)</span> be "not too close
to 0 or 1".</p>
<p>This modeling is the same as <a href="https://eprint.iacr.org/2011/487.pdf">prior work</a>, but we get better bounds
for <span class="math">\(\varepsilon\)</span> and <span class="math">\(\delta\)</span>. This is due to a neat trick: we can reuse the
results from <a href="https://arxiv.org/abs/1903.02837">amplification by shuffling</a>! The context is different,
but the underlying math is the same: take a bunch of small random things, mix
them together, and you get a good amount of randomness in total. That's
Theorem 1 in the <a href="https://arxiv.org/abs/1905.00650">paper</a>, if you want to follow along. And it can easily be
extended to settings where people are choosing among more than two options
(Corollary 1).</p>
<p>We continue by showing that <em>thresholding</em> can provide some guarantees, under
the right assumptions. Of course, thresholding only has an effect for <em>rare</em>
events: when all the <span class="math">\(p_i\)</span> are very small, and we only publish results if the
total number of <em>Yes</em> is larger than a threshold. Then, depending on the exact
parameters, we can get reasonably small values of <span class="math">\(\varepsilon\)</span> and <span class="math">\(\delta\)</span>.
Again, this captures an intuitive phenomenon: if very few people are in a group,
but we are suppressing this group from the results entirely, then this can
provide some protection (Theorem 4).</p>
<p>Putting these two results together, we get what we were looking for: a formal
interpretation of the privacy of <span class="math">\(k\)</span>-anonymity… Or, at least, of the simplest
possible <span class="math">\(k\)</span>-anonymity scheme, with a long list of assumptions. The high-level
intuition is as follows. For each group in the output:</p>
<ul>
<li>either there are many (more than <span class="math">\(k\)</span>) people in this group, and then the
attacker will have some uncertainty, because they don't know everyone there;</li>
<li>or there aren't that many people in the group, but then the count isn't
published, which protects the people in it.</li>
</ul>
<p>This is Theorem 5 in the <a href="https://arxiv.org/abs/1905.00650">paper</a>. Mission accomplished! Right?</p>
<p>Well… Not really. For this theorem to hold, we have to make a number of weird
assumptions. The "uncertainty distribution" <span class="math">\(\theta\)</span> has to have <em>exactly</em> the
right shape. There needs to be a "special group" whose count we don't publish,
regardless of the number of people in it. And as we'll see in the rest of this
article, the definition of privacy itself is more than a little shaky.</p>
<h2 id="a-distinction-between-active-and-passive-attackers">A distinction between active and passive attackers</h2>
<p>A first difficulty we identified is the necessity to distinguish between
<em>active</em> and <em>passive</em> attackers. When the attacker has partial knowledge of the
input data, this can mean two things.</p>
<ul>
<li>The attacker can inject some fake records in our data: they're <em>active</em>.</li>
<li>Or the attacker can get information on some records, but not influence them:
they're <em>passive</em>.</li>
</ul>
<p>Both situations can be realistic in different scenarios. For example, say you
call people on the phone to take part in a survey. Then, an attacker might
know some participants, but not be able to answer in their place. But what if
you're releasing statistics about the use of an online service? Then, an
attacker might be able to spin up bots to interact with it, and create
artificial data.</p>
<p>With "real" differential privacy, this doesn't matter. Giving the attacker
control over all the records doesn't change anything: you have to reason about
the worst-case scenario anyway, so you end up with the same definition.</p>
<p>But with partial knowledge, you do get two distinct versions of the definition,
depending on the attacker's capability. Some results, like the one about
thresholding, only hold in the passive version of the definition. That makes
sense: if the attacker can inject fake records, they can artificially boost low
numbers to make them go above the threshold. In this case, thresholding as a
privacy mitigation is pointless.</p>
<p>In fact, this concern isn't just theoretical. In a <a href="https://sciendo.com/article/10.2478/popets-2019-0013">paper investigating
Facebook's ad targeting system</a>, researchers used a similar method: by
carefully controlling the data and queries, they negated a threshold-based
protection.</p>
<h2 id="problems-with-dependencies-in-the-data">Problems with dependencies in the data</h2>
<p>Second, you might have raised your eyebrows at one of the hypotheses we made to
prove our initial results: all records had to be independent. This assumption
turns out to be both very important, and quite brittle.</p>
<ul>
<li>If we don't make this assumption, then the math gets difficult <em>fast</em>. You
need to find a way to model possible dependencies, and take them into account
when calculating the privacy guarantees. I'm not sure anybody has found a
tractable way to do this. I tried and failed to get anything convincing.</li>
<li>But if this assumption is false, and the attacker can use some knowledge about
the correlations in the data… Then everything breaks down. The privacy
guarantees no longer hold, and maybe the attacker can get all the data they
want.</li>
</ul>
<p>We also found an additional subtlety. When modeling the attacker's partial
knowledge, it is <em>crucial</em> to delineate what they know from what they want to
achieve. If there are correlations between their partial knowledge and the
sensitive information they try to find out… Then the definition is meaningless:
you end up in situations where an attacker with <em>more</em> knowledge over the data
is <em>less</em> powerful than one with less.</p>
<p>We're not the first to point out such problems: <a href="https://people.reed.edu/~agroce/FOCS2013.pdf">Bassily et al.</a> showed that
in certain cases, the original notion can be buggy, and proposed an alternative.
But we showed that this alternative can <em>also</em> have fundamental problems. This
separation between partial knowledge and sensitive information seems essential
to fix them.</p>
<p>This requirement, though, makes these definitions really tricky to use in
practice. We have to think hard about what the attacker wants to achieve, and
what they might know about the data. And if we're wrong, then everything we
proved might be meaningless.</p>
<h2 id="difficulties-making-composition-work">Difficulties making composition work</h2>
<p>Finally, these variants of differential privacy don't <a href="differential-privacy-awesomeness.html#composition">compose</a> very well.
Imagine a database of referendum votes, where each person is associated with
their vote. Then, consider the two following queries (without noise).</p>
<ul>
<li>The count of everybody who voted <em>Yes</em>.</li>
<li>The count of everybody who voted <em>Yes</em>, excluding a specific user <span class="math">\(i\)</span>.</li>
</ul>
<p>Each query, on its own, might be considered private in the partial knowledge
setting: if the attacker has uncertainty over many votes, each query result
won't give a lot of information about individual votes. But, of course,
combining both results gives you the vote of user <span class="math">\(i\)</span>. So, composition doesn't
degrade privacy loss smoothly, like for differential privacy. Instead, it can
lead very quickly to catastrophic privacy loss.</p>
<p>Of course, this counterexample is a little artificial. It's natural to wonder:
can we find a simple condition on queries allowing us to prove a composition
result? We investigated, and found possible ways of achieving composition… But
nothing was really natural and convincing.</p>
<p>The problem is, again, correlations: if queries give correlated results, then
the math breaks down. That's bad news: the queries can't touch the same data,
otherwise, the results are correlated. What if the queries look at different
columns of a dataset? You need one more assumption: the different columns in the
data must be uncorrelated. And that's pretty unrealistic.</p>
<p>One option is a little more viable: databases that constantly get new data. This
situation is common in practical scenarios. And it seems natural to assume that
this new data might <em>add</em> to the attacker's uncertainty… So, if we require that
each new query gets some "fresh" data, we can get some composition results.
They're quite preliminary for now, but maybe worth investigating further.</p>
<h1 id="conclusion-perspectives">Conclusion & perspectives</h1>
<p>I'm happy about some of the progress we made on this problem. We found important
issues with prior work, and proposed a more robust definitional framework. The
link we established with shuffled DP is interesting, and somewhat promising. I'm
hopeful that both aspects might end up being useful to folks doing further
research in this area. </p>
<p>But while the promise of utilizing an attacker's uncertainty in DP is alluring,
a closer look revealed big challenges with this line of work. I'm not sure I'm
optimistic about making this idea work well enough for real-world use cases. The
assumptions that are necessary to make this work are too brittle and
unrealistic. The math seems to get messy too fast.</p>
<p>While I was working on this research project, I was also working on building
infrastructure for differential privacy. And over time, I became convinced that
the gap between DP theory and practice was <em>usability</em>. We'll get to widespread
adoption by building better tools and doing a better job helping people use
them. I even made a whole <a href="bottom-up-differential-privacy.html">talk</a> about this idea since then.</p>
<p>This work also changed my mind about the assumptions behind differential
privacy. They're not unrealistic, or too strong: instead, they seem necessary to
get the basic properties you want out of a convincing notion of privacy. They
might be an over-approximation of the capabilities of realistic attackers… But
reducing these assumptions is dangerous, costs a lot, and doesn't buy much.</p>
<p>If you've found this post interesting, or if you disagree with some of the
points I made in it, let me know! My contact info is at the footer of this page,
and I'd love to hear more perspectives about this problem.</p>
<hr>
<p><small>Thanks to <a href="https://mohammadi.eu/">Esfandiar Mohammadi</a> and <a href="https://people.inf.ethz.ch/basin/">David Basin</a> for
their helpful feedback on drafts of this post.</small></p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>A bottom-up approach to making differential privacy ubiquitous2022-03-09T00:00:00+01:002022-03-09T00:00:00+01:00Damien Desfontainestag:desfontain.es,2022-03-09:/privacy/bottom-up-differential-privacy.html<p>This post contains the slides and speaker notes for an invited talk I delivered at PPAI-22.</p><p>This post is a transcript of an invited talk I delivered to
<a href="https://aaai-ppai22.github.io/">PPAI-22</a>. It was also published on the <a href="https://www.tmlt.io/resources">Tumult
Labs website</a>. <a href="https://users.cs.duke.edu/~ashwin/">Ashwin Machanavajjhala</a>, <a href="https://people.cs.umass.edu/~miklau/">Gerome
Miklau</a>, <a href="https://www.linkedin.com/in/philip-bohannon-88624a2">Philip Bohannon</a>, and <a href="https://www.linkedin.com/in/samuel-haney-47a16819b">Sam Haney</a> contributed to
these slides.</p>
<hr>
<p>Hi everybody! Here is a graph counting the number of academic papers related to
differential privacy, over time.</p>
<p><center>
<img alt="Line graph labeled "year" on the horizontal axis, and "publications" on the
vertical axis. It goes up from 0 in 2006 to approximately 4300 in
2021." src="https://desfontain.es/privacy/images/ppai-22-talk-00.png">
</center> </p>
<p>In academia, differential privacy essentially won. There is broad agreement, at
least among computer scientists, that this is the notion of choice to formally
bound the privacy leakage when publishing data. Differential privacy has become
the default tool that people use to quantify trade-offs between privacy and
accuracy.</p>
<p>The field is growing every year, with exciting new domains of application,
empirical improvements, and theoretical advances.</p>
<p>For comparison, here is a graph showing the number of <a href="real-world-differential-privacy.html">real-world
deployments</a> that I could find public information about. </p>
<p><center>
<img alt="The same graph, except the vertical axis is labeled "deployments", and the line is completely flat, at 0." src="https://desfontain.es/privacy/images/ppai-22-talk-01.png">
</center> </p>
<p>As you can see… Ooops! Sorry. I forgot to change the scale of the vertical axis.</p>
<p><center>
<img alt="The same graph, but the vertical axis now goes from 0 to 10. Besides a single
point at 1 in 2008, the line starts going up in 2016, and reaches 6 in
2021." src="https://desfontain.es/privacy/images/ppai-22-talk-02.png">
</center> </p>
<p>OK, now we’re seeing something… It’s not much, though. We’re still at a stage
where I can list all public deployments of differential privacy in a <a href="real-world-differential-privacy.html">single
blog post</a>.</p>
<p>I know what you’re going to say, though. There might not be many use cases, but
some of these are <em>really big</em>. </p>
<p><center>
<img alt="A slide containing the logos for the following organizations: the U.S. Census
Bureau, Google, Facebook, Microsoft, LinkedIn, the Internal Revenue Service,
Appl, and OhmConnect," src="https://desfontain.es/privacy/images/ppai-22-talk-03.png">
</center> </p>
<p>The <a href="https://www.census.gov/programs-surveys/decennial-census/decade/2020/planning-management/process/disclosure-avoidance.html">2020 Decennial Census</a>! Mobility data from around the globe to
<a href="https://arxiv.org/abs/2004.04145">help combat COVID-19</a>! <a href="https://www.microsoft.com/en-us/research/publication/collecting-telemetry-data-privately/">Telemetry collection</a> from <a href="https://docs-assets.developer.apple.com/ml-research/papers/learning-with-privacy-at-scale.pdf">billions of
devices</a>!</p>
<p>If we look at who is deploying DP, one thing in common for almost all of these
organizations is that they’re <em>large</em>. They can afford to invest in, or contract
with, specialized science and engineering teams to help them roll out this
technology.</p>
<p>Special mention to <a href="https://www.ohmconnect.com/">OhmConnect</a>, the only exception I could find to
this rule! They’re a startup sharing <a href="https://assets.website-files.com/5cb0a177570549b5f11b9550/5ffddb83b5ea5d67f5c43661_Quantifying%20The%20OhmConnect%20Virtual%20Power%20Plant%20During%20the%20California%20Blackouts.pdf">smart meter data</a> to increase
power grid reliability.</p>
<p>But the problem that differential privacy solves isn’t limited to these
massively large organizations: smaller organizations also have data sharing and
publishing needs! Everyone could benefit from using strong anonymization
techniques, not just these giants.</p>
<p>This is the question I’m here to talk about today. How do we bridge that gap?
How do we make differential privacy ubiquitous?</p>
<p><center>
<img alt="The introductory slide of a talk titled "A bottom-up approach to making
differential privacy ubiquitous". The slide contains author information (Damien
Desfontaines, @TedOnPrivacy), affiliation (the Tumult Labs logo), and a
copyright notice. The text is in white, the image behind is a series of
perturbed purple lines on a black
background." src="https://desfontain.es/privacy/images/ppai-22-talk-04.png">
</center> </p>
<p>I’m <a href="/serious.html">Damien</a>, and I work as a scientist for <a href="tmlt.io">Tumult
Labs</a>. We’re a startup trying to make widespread adoption of
differential privacy into a reality.</p>
<p>In this presentation, I’ll outline a <em>bottom-up</em> approach for reaching that
goal.</p>
<p>What do I mean by “bottom-up”? Well, first, here’s what a “top-down” approach
could look like.</p>
<p><center>
<img alt="A slide split in two. On the left, the title is "Top-down", and lists three
bullet points: "Lobby decision makers", "Get DP into standards, laws, internal
best practices…", and "Adoption follows". On the right, the slide is
empty." src="https://desfontain.es/privacy/images/ppai-22-talk-05-01.png">
</center> </p>
<ul>
<li>First, we lobby decision-makers: we convince executives, regulators, standard
committees, etc., that differential privacy should be the notion of choice for
anonymizing data.</li>
<li>Then, once DP has become a requirement in different places, like standards,
regulations, internal best practices, etc.…</li>
<li>People adopt it because they have to.</li>
</ul>
<p>This comes with many challenges.</p>
<ul>
<li>First, writing good policy documents and guidance is very difficult. We would
need to answer questions like “how to choose parameters”, which are already
tricky for specific use cases, but even harder to decide on in generic terms.</li>
<li>Second, people won’t sign off on a technology unless they’re convinced it can
work in practice. Differential privacy needs to prove itself in the field, in
sufficiently many cases, in each vertical it can be applied to, before that
happens.</li>
<li>Finally, when privacy/security efforts are compliance-oriented, implementation
can often be people doing the bare minimum. That might not be too great.</li>
</ul>
<p>To be clear: this outreach work with key decision-makers is still valuable, and
worth doing! But this isn’t what we’re focusing on in the immediate future.</p>
<p>Instead, we’re pushing for a bottom-up approach.</p>
<p><center>
<img alt="The same slide as before, with the right part filled in. The title is
"Bottom-up", and lists three bullet points: "Get data scientists excited",
"Train them to become DP practitioners and advocates", and "Make deployment
super easy"." src="https://desfontain.es/privacy/images/ppai-22-talk-05-02.png">
</center> </p>
<ul>
<li>First, we create excitement among data scientists and engineers, across the
industry.</li>
<li>Then, we create a well-lit path that makes it super easy for these people to
go from “I’m curious what this technology can do” to “I know exactly how it
will work for my use case!”</li>
<li>And we create tools to make this entire process super easy, all the way to
deployment.</li>
</ul>
<p>Sign-off from decision-makers happens at the end of the process, not at the
beginning. By that time, all they need to do is confirm that it works.</p>
<p>Note that as far as I know, this is what happened for most of the deployments
that I could find described publicly. There wasn’t an executive giving the order
to use differential privacy, out of the blue. Instead, individual teams of
engineers and scientists built prototypes, showed that it worked in practice,
did internal advocacy, and eventually got the go-ahead.</p>
<p>So, how do enable many more people and organizations to get to this point?</p>
<p>We have a vision, and we need your help.</p>
<p>We said that the way to get adoption is to make usable tools for differential
privacy, and train people to use them. By the end, we want thousands of
engineers and data analysts to become DP practitioners. What does that learning
path look like in practice?</p>
<p><center>
<img alt="A slide containing a horizontal arrow labeled "Time invested / level of
expertise gained.", with 8 boxes describing different points on the line:
"Downloading and installing an open-source library", "Following tutorials",
"Generating DP data for the first time", "Iterating to optimize privacy-accuracy
tradeoffs", "Selecting parameters", "Getting sign-off from decision makers",
"Figuring out operational issues, preparing to deploy", and a rocket ship
emoji." src="https://desfontain.es/privacy/images/ppai-22-talk-06.png">
</center> </p>
<ul>
<li>First, people might hear about differential privacy, and decide to give it a
try, using an open-source tool.</li>
<li>They’ll follow tutorials to get the hang of it…</li>
<li>… and maybe reach a point where they’re giving it a first try on their own
data.</li>
<li>Then, they’ll probably need to optimize privacy-accuracy trade-offs…</li>
<li>… and if they’re convinced that this is workable, start thinking of which
parameters would make sense for their use case.</li>
<li>Once they get the sign-off from their hierarchy…</li>
<li>… they will need to do a bunch of operational deployment stuff …</li>
<li>… and end up shipping a differentially private data release.</li>
</ul>
<p><center>
<img alt="A graph where the horizontal axis is labeled "Time invested / level of
expertise gained.", the vertical axis is labeled "Number of users", and a dashed
vertical line near the right of the graph is labeled "Expertise necessary to
ship a given production use case". The OhmConnect logo is on the right size of
this dashed line." src="https://desfontain.es/privacy/images/ppai-22-talk-07.png">
</center> </p>
<p>Our goal is to get people to that point on the right, where they can deploy DP
to production.</p>
<p>As I mentioned earlier, I could find one small company that went further than
this line. We want to get to many more such examples.</p>
<p>In fact, we’re going to visually represent how many people there are at each
step of the process.</p>
<p><center>
<img alt="The same graph as earlier, with a line starting halfway to the vertical axis,
and going down in cliffs, reaching the horizontal line before the "shipping to
production" indicator. Each cliff is labeled: "I don’t understand half the words
in this interface & docs!", "What are all these new parameters I need to
specify? I’ve never needed that in SQL!", "Wow, my initial results are basically
pure noise. What do I do about it?!", and "I can’t predict nor explain how
accurate my output data is going to be. I can’t ship
this!"" src="https://desfontain.es/privacy/images/ppai-22-talk-08.png">
</center> </p>
<p>Today, the curve might look like this. Even though there might be some initial
interest, almost nobody ends up crossing that line we’re interested in. So why
is that?</p>
<p>We don’t know the answer for sure, but we’ve heard of a lot of hurdles that
people encounter when trying to roll out differential privacy. Here are a few of
them.</p>
<ul>
<li>First off, people might be immediately put off by how complex tooling looks
like. If the interface & documentation looks like it was designed for people
who already know what they’re doing, they might simply think “OK, I’m not the
target audience”, and give up before even trying it out.</li>
<li>Second, DP comes with additional requirements: things like group-by keys, or
clamping bounds, are new concepts that people never had to think about before.
People might think: why is it so hard to do even basic things that would take
me 3 lines in SQL? And drop off, thinking that it’s just going to get worse
from there.</li>
<li>If people reach the point of trying it out on their own data, the initial
results might be absolute garbage, because the strategy is extremely
sub-optimal. This can be demoralizing, and make people feel like they won't
ever make this work.</li>
<li>Even if the results end up looking kind of reasonable when plotted on a graph,
this might not be enough. People need stronger guarantees on how accurate the
data is, and if the tool doesn’t provide this, this might also be a hard
blocker.</li>
</ul>
<p><center>
<img alt="The same graph as earlier, but this time, the line starts higher, and only
goes down a little, before crossing the "deployment" line a little higher than
half-way. A circle marks the intersection
point." src="https://desfontain.es/privacy/images/ppai-22-talk-09.png">
</center> </p>
<p>Instead, this is where we want to be. We won’t ever make the line entirely flat,
that’s normal, every software project loses users in the learning process. But
our goal is to maximize the number of people reaching deployment. We want the
intersection point to be as high as possible.</p>
<p>OK, so how do we do it?</p>
<p><center>
<img alt="The same graph as earlier, with an arrow pointing up next to the beginning of
the line. This early stage is labeled "Marketing efforts, blog posts,
outreach…"." src="https://desfontain.es/privacy/images/ppai-22-talk-10.png">
</center> </p>
<p>First, we make the line start higher. We want to get as many people as we can who know that differential privacy exists, and have an idea of what problem it solves. Further than that, we want to get them excited about trying it out.</p>
<p>Some of this is the job of companies like mine: we need to do a good job at
marketing this technology.</p>
<p>But researchers can also help there: we need many more resources that are
accessible to beginners, like blog posts! These serve the dual purpose of
helping people learn, and of making our field more widely known to the public.</p>
<p>We also need y’all to contribute to the public discussions around data privacy,
beyond research papers. There can be many examples of that kind of work.</p>
<ul>
<li>Opinion pieces in scientific publications, like the ACM magazines, or in
newspapers, can do wonders to raise awareness.</li>
<li>Participating to events or workshops discussing adjacent problems can be great
to open your research horizons, and to socialize with people outside of your
usual research community. Attending non-academic events around data privacy,
in particular, can be eye-opening.</li>
<li>Finally, getting into the contact list of a tech journalist whose work you
follow is easier than you think! Reach out and let them know that you’re happy
to comment on technical topics in your area of expertise, and they’ll be happy
to take you up on the offer some time.</li>
</ul>
<p>All of these can have a major impact. Thanks to everyone who is already doing
this kind of work today! We need even more.</p>
<p>OK, once we made that line start as high as we could, what do we do next?</p>
<p><center>
<img alt="The same graph as earlier, with a dashed angle showing that the line must
initially go down not too fast. This is labeled "Simple interfaces, great
learning docs"" src="https://desfontain.es/privacy/images/ppai-22-talk-11.png">
</center> </p>
<p>We make sure that we lose as few people as we can in the initial learning
stages. We make that curve as flat as possible, avoiding those cliffs from
earlier. We make the learning process as smooth as we can.</p>
<p>To do that, we need interfaces that are super simple to use, and a great
onboarding experience. Let me give you a sneak peek of what the interface looks
like on our platform right now.</p>
<p><center>
<img alt="A Python code snippet.
session = Session.from_dataframe(
dataframe=private_data,
source_id="my_data",
privacy_budget=PureDPBudget(1.7),
)
query = (
QueryBuilder("my_data")
.filter("age > 42")
.groupby(zip_codes)
.median("income", low=0, high=10**6)
)" src="https://desfontain.es/privacy/images/ppai-22-talk-12.png">
</center> </p>
<p>Our platform is built in Python, and runs on top of Spark, so we can scale to
very large datasets.</p>
<ol>
<li>To use it, you start by defining a <em>session</em>. This session encapsulates your
data, given as a Spark dataframe, and gives you clear privacy guarantees.
Here, the library promises you that everything downstream of this session
will satisfy differential privacy, with ε=1.7.</li>
<li>Then, you write a <em>query</em>, using a Spark-like query language. Here, the query
filters the records to only keep the individuals older than 42, then we group
by zip codes, and we compute the median income for each zip code.</li>
<li>Finally, you <em>evaluate</em> the query using a portion of your privacy budget,
here, 0.8. The result is a regular Spark dataframe. We could, later, evaluate
further queries, as long as we don’t spend more budget than was initially
allocated.</li>
</ol>
<p>There are a couple of things that will still seem unfamiliar to data scientists
without prior experience with differential privacy: the way we specify group-by
keys, for example, or clamping bounds. We’re working hard to make these even
simpler and more accessible.</p>
<p>That’s what we’re doing. Now, what can the academic community do to help users
in this initial learning stage?</p>
<p><center>
<img alt="The same graph as earlier, with the dashed angle. This time, the label
shows "usable algorithms" in bold, in addition to "Simple interfaces, great
learning docs"." src="https://desfontain.es/privacy/images/ppai-22-talk-13.png">
</center> </p>
<p>One way is by thinking about usability when designing new techniques to achieve
DP. How many choices will a user have to think about before using a given
mechanism?</p>
<p>Suppose, for example, that you found a novel a way to compute quantiles. It’s
better than the state of the art, but it introduces new hyperparameters: for
example, you need to discretize the data first, and the user can choose the
granularity. Can you recommend a good default for this new parameter? If there
is no universally reasonable choice, can you automatically and privately select
it based on the data, using some portion of the budget?</p>
<p>Doing this will make it much more likely that people can use your fancy
algorithms, even in the early stages of the process.</p>
<p>Ok, so that was the initial learning stage. What comes next?</p>
<p><center>
<img alt="The same graph as earlier, with the dashed angle further on the right,
closer to the deployment line, indicating that the line should become flat. The
label says "advanced features"." src="https://desfontain.es/privacy/images/ppai-22-talk-14.png">
</center> </p>
<p>As people start moving out of the learning phase and closer to the deployment
stage, we need to keep supporting them. Again, we want that curve to stay flat.
We don’t want people to drop off just before the finish line.</p>
<p>Once people start trying to ship their initial idea to production, they start
having more complex needs that require advanced features. For example, in one of
the data releases we’re working on with the US Census Bureau, the goal is to
publish statistics on various population groups, depending on geographic and
racial characteristics. These groups can vary tremendously in size: in some
cases, we can only publish total counts with reasonable accuracy, while in
others, we also want to split these groups into more fined-grained categories.</p>
<p>This requires an adaptive algorithm. Let me show you what it looks like in our
interface. It’s going to be little more complicated than the previous example,
but don't worry, I’ll walk you through it step by step.</p>
<p><center>
<img alt="Three code snippets with accompanying visuals. //
budget_10 = RhoZCDP(total_budget / 10.)
budget_90 = RhoZCDP(total_budget * 9./10.)
This is represented by a pie chart splitting a disc in 1/10 and 9/10. //
histogram = session.evaluate(
QueryBuilder("data")
.groupby(geo_races)
.count(),
privacy_budget=budget_10,
)
This is represented by a histogram, and uses the 1/10 part of the privacy budget
pie. //
hist_with_category = histogram.withColumn(
"under_threshold",
"IF(count < 42, ‘true', 'false')",
)
This is represented by the same histogram, with a horizontal dashed line
determining whether each bucket is above and below, and marking it with
different colors depending." src="https://desfontain.es/privacy/images/ppai-22-talk-15.png">
</center> </p>
<p>First, we set aside 10% of our total budget. Here, we use zero-concentrated DP,
because each individual will contribute to many statistics, so we’re using
Gaussian noise and tight privacy accounting methods. Note that here, switching
to another privacy definition is as simple as changing the privacy budget type:
the framework is extensible enough to make this kind of operation very easy.</p>
<p>We then use that budget to compute, for each population group, a total count of
people. Here, we group by geography and race/ethnicity combinations.</p>
<p>Then, we augment these results by checking, for each of these groups, whether
the count is below or above a certain threshold. Later on, we will want to do
different things depending on the value of this column.</p>
<p><center>
<img alt="Two code snippets with accompanying visuals. //
session.create_view(
QueryBuilder("data")
.join_public(hist_with_category),
"data_with_category",
)
This is represented by the same histogram split in two as earlier. //
budget_10 = RhoZCDP(total_budget / 10.)
budget_90 = RhoZCDP(total_budget * 9./10.)
This is represented by a pie chart splitting a disc in 1/10 and 9/10.
new_sessions = session.partition_and_create(
"data_with_category",
privacy_budget=budget_90,
attr_name="under_threshold",
splits={
"total": "true",
"detail": "false"
},
)
This is represented by two arrows leading to two separate histograms, one with
the buckets above the threshold, one with the buckets under the
threshold. This uses the 9/10 part of the privacy budget
pie." src="https://desfontain.es/privacy/images/ppai-22-talk-16.png">
</center> </p>
<p>At this point, we have a table that tell us, for each group, whether the noisy
count of people is above or below a threshold.</p>
<ul>
<li>We join our secret data with this augmented table. Each
individual record is now associated with additional information telling us
whether the group they’re a part of has a count below or above the threshold.</li>
<li>And once we have this new, augmented private data set, we partition the
session into two sessions, depending on the value of this column. One session
will have part of the data, the other will have the rest. These sessions are
allocated a given privacy budget; here, we use the entirety of the privacy
budget we have left. That budget is depleted from the original session, and
transferred to the new sessions.</li>
</ul>
<p><center>
<img alt="Two code snippets with accompanying visuals. //
total_session = new_sessions["total"]
total_counts = total_session.evaluate(
QueryBuilder("data_with_category")
.groupby(geo_races)
.count(),
privacy_budget=budget_90,
)
This is represented by the smaller histogram changing a little bit, using the
9/10 part of the budget. //
detail_session = new_session["detail"]
detail_counts = detail_counts.evaluate(
QueryBuilder("data_with_category")
.groupby(geo_races * age)
.count(),
privacy_budget=budget_90,
)
This is represented by the larger histogram, where each bucket is split in three
sub-buckets, using the 9/10 part of the
budget." src="https://desfontain.es/privacy/images/ppai-22-talk-17.png">
</center> </p>
<p>Now, we have two sessions, so we can do different things in each.</p>
<ul>
<li>For the records that are in small groups, we only compute the total counts,
with the rest of the privacy budget. This is the same aggregation as earlier,
but with more precise results.</li>
<li>And for the records that are in bigger groups, we compute the counts at a more
granular level, also including age. Here, the multiplication operator in the
groupby correspond to doing the cross-product of groupby keys for different
attributes.</li>
</ul>
<p>Of course, the actual algorithm is a lot more complicated. We actually split the
data in four different levels of granularity, so we have three different
thresholds, ages are bucketed, we use more demographic information, and we
compute a lot more things. But hopefully, this gives you an idea of what
real-world use cases can require, and what kind of advanced features our
platform can support.</p>
<p>OK, so that’s an example of what we’re doing to support people at this stage.
Can the academic community also help flattening this curve, and make it easy for
people to reach deployment?</p>
<p><center>
<img alt="The same graph as before the code snippets, with the dashed angle at the same
place as before. The label now says "and so. many. open problems." in addition
to "advanced features"." src="https://desfontain.es/privacy/images/ppai-22-talk-18.png">
</center> </p>
<p>Yes. Yes, yes, yes. In this path towards deployment, there are so many open
problems. People routinely need things that don’t exist yet. Our customers are
constantly asking us for very reasonable things, natural requests that… turn out
to be open science problems. This is why we’re <a href="https://tmlt.io/careers">hiring scientists</a>, by
the way.</p>
<p>Let me give you a distilled list of areas where we desperately need more
progress in research and engineering.</p>
<p><center>
<img alt="A slide titled "Open problems", listing four areas: "Explainability /
Transparency", "Decision support", "Fitness-for-use", and "Operational
challenges"." src="https://desfontain.es/privacy/images/ppai-22-talk-19.png">
</center> </p>
<p>First, explainability and transparency. Releasing private error measures along
with the DP output is easy for simple mechanisms, but still hard for things like
quantiles, or when clamping bounds are involved. More generally, can we explain
to non-expert users what was done to the data? Can we give them a summary they
can understand and use in later analyses? Finally, DP algorithms can introduce
biases in the data – can we make these transparent, and allow data users to take
them into account? Explainability and transparency are absolutely critical to
build trust, and trust is key to adoption. We, as a field, need to have better
answers for these questions.</p>
<p>A second one is decision support tools. Dashboards and visualizations that allow
people to understand the privacy/accuracy trade-offs in their data, and
fine-tune parameters, are critical. In our experience, this is often what makes
people “get it”, and make them feel like they can actually use this tech. This
is a promising area of research for visualization and usable privacy folks, but
there are also complicated algorithmic questions here: how do we do that
efficiently?</p>
<p>When people want to generate DP data, they don’t want to specify a budget:
instead, it would be much nicer if they could decide what level of data quality
is fit-for-use – good enough for their use case – and specify that as input to
the algorithm. Note that these data quality measures are often interpreted as
the error of a single noisy estimate or parameter. But in real use cases, it can
be a lot more complex: for example, will the relative ranking of items based on
noisy estimates be approximately correct?</p>
<p>Finally, operational aspects of DP are critical. How do we keep a good
accounting of the privacy loss over time, for data releases that happen every
day or week? How can we validate that the DP data is correct before publishing
it? How should we handle failures? How do we detect drifts in accuracy, and how
should we handle these alerts? Work on these topics is starting to emerge in
academia, but there is a lot more to be done.</p>
<p><center>
<img alt="The same graph as before, but the dashed angle has been replaced with an arrow
starting at the dashed deployment line, and pointing to the left. It is labeled
"More data!"." src="https://desfontain.es/privacy/images/ppai-22-talk-20.png">
</center> </p>
<p>Back to our curve. One last thing we can do is lowering the level of expertise
necessary to ship DP to production is as small as possible. The faster people
get there, the less likely they’re going to drop off.</p>
<p>This first requires more data: what do people actually need to do? Once we know,
we can build the advanced features that people need, and build interfaces that
make them easier to use.</p>
<p>The call to action here is: if you know of more people using differential
privacy in practice, try convincing them to communicate about this! Even when
there isn’t novel science involved, it’s still worth telling the world what you
did, and ideally, why you did it. This way, we can know what problems people
encounter in practice, and what are the most pressing issues to solve to
increase adoption. As a an added bonus, communicating about your use case for
differential privacy is a great way to foster trust among stakeholders, and to
convince other people to also try using DP!</p>
<p><center>
<img alt="A slide split in two, titled "Two more ways to help". On the left, it says
"Beta-test the Tumult Platform!", and links to tmlt.io/connect. On the right,
"Join our team!", with a link to
tmlt.io/careers." src="https://desfontain.es/privacy/images/ppai-22-talk-21.png">
</center> </p>
<p>There are two more ways you can help.</p>
<ul>
<li>If the little code snippets I showed sounded interesting, and you’d like to
play with our platform and give us feedback, <a href="https://tmlt.io/connect">let us know</a>! We’re
happy to give you a preview before our open-source launch later this year.</li>
<li>Finally, if you’d like to apply your research skills to hard, impactful
real-world problems, and work with a great team, <a href="https://tmlt.io/careers">drop us a line</a>!
We’re looking for scientists in Europe and in the US.</li>
</ul>
<p><center>
<img alt="An outro slide, saying "Thanks ♥", giving displaying author information, the
Tumult Labs logo, and the two links from the previous
slide." src="https://desfontain.es/privacy/images/ppai-22-talk-22.png">
</center> </p>
<p>Thanks again for the invitation and for attending this presentation! I'm looking
forward to your questions, and I'm also happy to continue the conversation by
email or via Twitter.</p>Averaging risk: Rényi DP & zero-concentrated DP2022-02-20T00:00:00+01:002022-11-13T00:00:00+01:00Damien Desfontainestag:desfontain.es,2022-02-20:/privacy/renyi-dp-zero-concentrated-dp.html<p>Averaging the privacy loss random variable across outputs can be useful: introducing Rényi DP, and zero-concentrated DP.</p><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p></small></p>
<hr>
<p><span class='lettrine'>R</span><strong>emember</strong> the <a href="privacy-loss-random-variable.html">privacy loss random
variable</a> (PLRV)? We saw that it described all values that the privacy
leakage could take, and how likely each one was. And we saw that
<span class="math">\(\varepsilon\)</span>-DP was a <em>worst-case</em> property: the PLRV must <em>always</em> be lower
than <span class="math">\(\varepsilon\)</span>. By contrast, we described <span class="math">\((\varepsilon,\delta)\)</span>-DP as an
"almost worst-case" notion: it left a little wiggle room for rare events with
a privacy loss larger than <span class="math">\(\varepsilon\)</span>.</p>
<p>In this article, we'll use the PLRV in yet another interesting way. Instead of
only looking at its extreme values, we'll look at the <em>average</em> of the PLRV.
This will give us an intuitive explanation of two useful variants of DP: <em>Rényi
differential privacy</em>, and <em>zero-concentrated differential privacy</em>.</p>
<h1 id="basic-idea">Basic idea</h1>
<p>Recall the core intuition behind the privacy loss random variable (PLRV). Say
our secret mechanism <span class="math">\(A\)</span> returns an output <span class="math">\(O\)</span>. The attacker is trying to find
out whether the input was <span class="math">\(D_1\)</span> or <span class="math">\(D_2\)</span>, where both options differ in a single
person's data. The PLRV <span class="math">\(\mathcal{L}_{D_1,D_2}(O)\)</span> was kind of the
"<a href="privacy-loss-random-variable.html">actual</a> <span class="math">\(\varepsilon\)</span> value" for this attacker:
<span class="math">\(e^{\mathcal{L}_{D_1,D_2}(O)}\)</span> was the advantage they gain when observing output
<span class="math">\(O\)</span>. This gave us a reformulation of <span class="math">\(\varepsilon\)</span>-DP, as an <em>upper bound</em> on
this value. If <span class="math">\(A\)</span> is <span class="math">\(\varepsilon\)</span>-DP, then for all possible choices for <span class="math">\(D_1\)</span>
and <span class="math">\(D_2\)</span>, and all possible outputs <span class="math">\(O\)</span>:</p>
<div class="math">$$
\mathcal{L}_{D_1,D_2}(O)\le\varepsilon.
$$</div>
<p>How do we transform this worst-case property into an average case definition?
Two options come to mind.</p>
<ol>
<li>We could average the privacy loss across possible databases <span class="math">\(D_1\)</span> and <span class="math">\(D_2\)</span>…</li>
<li>… or we could average it across possible outputs <span class="math">\(O\)</span>.</li>
</ol>
<p>The first option turns out to be a Bad Idea™, for subtle reasons that I'm not
going to go into here<sup id="fnref:partial"><a class="footnote-ref" href="#fn:partial">1</a></sup>. The second option, however, makes sense. This
is the same "kind" of relaxation as <span class="math">\((\varepsilon,\delta)\)</span>-DP: there's a small
probability that something goes worse than we'd like. Importantly, that this
probability <em>doesn't depend on the attacker</em>. It only comes from the algorithm's
randomness, and doesn't require additional assumptions.</p>
<p>There is a significant difference, however. In <span class="math">\((\varepsilon,\delta)\)</span>-DP, there
can be a small probability (at most <span class="math">\(\delta\)</span>) of <em>infinite</em> privacy loss. When
we're averaging the privacy loss, that's no longer allowed. If the privacy loss
is very low 99.99% of the time, but is infinite with probability 00.01%… then
its average is still going to be infinite.</p>
<p>So bounding the average PLRV will be a way of relaxing DP, but without allowing
infinitely bad events. <em>Arbitrarily</em> bad events can still happen, but only with
vanishing probability. Let's formalize this.</p>
<h1 id="renyi-differential-privacy">Rényi differential privacy</h1>
<p>Here's a first attempt at capturing this intuition of averaging risk. For every
pair of databases <span class="math">\(D_1\)</span> and <span class="math">\(D_2\)</span> differing in a single record, we require that:</p>
<div class="math">$$
\underset{O\sim A(D_1)}{\mathbb{E}} \left[\mathcal{L}_{D_1,D_2}(O)\right]\le\varepsilon.
$$</div>
<p>Here, <span class="math">\(\mathbb{E}\)</span> is the <a href="https://en.wikipedia.org/wiki/Expected_value"><em>expected value</em></a>: you weigh each possible event
by its probability. A very bad event can be acceptable if it happens almost
never. This seems to capture our initial intuition.</p>
<p>One problem with this approach, though, is that we're not really averaging the
right thing. The advantage that a <a href="differential-privacy-in-more-detail.html#quantifying">Bayesian attacker</a> can gain is
<span class="math">\(e^\varepsilon\)</span>, not <span class="math">\(\varepsilon\)</span>! So by averaging the privacy loss, we're not
really averaging the risk. Let's show this with a little visualization. The
following plot shows the attacker gain of a made-up mechanism: how much will the
<a href="differential-privacy-in-more-detail.html#quantifying">betting odds</a> change, depending on the random output?</p>
<p><center class="largeimg">
<img alt="Graph showing a made-up distribution for the attacker's gain. With 30% probability, it has value 150, then decreases in a discrete, but exponential, manner." src="https://desfontain.es/privacy/images/example-exp-plrv.svg">
</center> </p>
<p>With this distribution, the expected value of <span class="math">\(\mathcal{L}\)</span> (the "average
<span class="math">\(\varepsilon\)</span>") is around <span class="math">\(3.4\)</span>. This translates to an attacker gain of about
30.</p>
<p><center class="largeimg">
<img alt="Same graph as above, with a red dashed line at y=30. It looks too low to represent the average value." src="https://desfontain.es/privacy/images/average-plrv.svg">
</center> </p>
<p>That doesn't look right. The actual risk is often a <em>lot</em> larger the "average"!
If we want to average out the risk, we should take the mean of <span class="math">\(e^\mathcal{L}\)</span>
instead. This would be closer to the intuition. The requirement would become…</p>
<div class="math">$$
\underset{O\sim A(D_1)}{\mathbb{E}} \left[e^{\mathcal{L}_{D_1,D_2}(O)}\right] \le e^\varepsilon.
$$</div>
<p>This makes more sense: this can be seen as the arithmetic mean of the risk. If
we plot it, the corresponding <span class="math">\(e^\varepsilon\)</span> is the average of the blue line.</p>
<p><center class="largeimg">
<img alt="Graph with the made-up distribution, with a red dashed line at y=68.5" src="https://desfontain.es/privacy/images/average-exp-plrv.svg">
</center> </p>
<p>This still feels a bit arbitrary. Why not take a different averaging function?
Large values of the privacy loss correspond to the worst events. These are
particularly scary: maybe we want to give them more "weight"? We could do this
using a <a href="https://en.wikipedia.org/wiki/Root_mean_square">quadratic mean</a>, for
example. We would then require something like this:</p>
<div class="math">$$
\underset{O\sim A(D_1)}{\mathbb{E}} \left[\left(e^{\mathcal{L}_{D_1,D_2}(O)}\right)^2\right] \le \left(e^\varepsilon\right)^2.
$$</div>
<p>This gives us a larger average than before.</p>
<p><center class="largeimg">
<img alt="Same graph as before, with a red dashed line at y=90.4" src="https://desfontain.es/privacy/images/average-square-exp-plrv.svg">
</center> </p>
<p>Let's generalize this. To decide <a href="https://en.wikipedia.org/wiki/Generalized_mean">which kind of averaging
function</a> to use, we'll
introduce a parameter <span class="math">\(\alpha\)</span>.</p>
<div class="math">$$
\underset{O\sim A(D_1)}{\mathbb{E}} \left[\left(e^{\mathcal{L}_{D_1,D_2}(O)}\right)^{(\alpha-1)}\right] \le \left(e^\varepsilon\right)^{(\alpha-1)}.
$$</div>
<p>This is <em>Rényi differential privacy</em>. If <span class="math">\(A\)</span> satisfies the above inequality for
all choices of <span class="math">\(D_1\)</span> and <span class="math">\(D_2\)</span>, we say it's <span class="math">\((\alpha,\varepsilon)\)</span>-Rényi
differentially private.</p>
<p>Some special values of <span class="math">\(\alpha\)</span> correspond to common averaging functions.</p>
<ul>
<li><span class="math">\(\alpha\rightarrow1\)</span> bounds the arithmetic mean of <span class="math">\(\mathcal{L}\)</span> or,
equivalently, the <a href="https://en.wikipedia.org/wiki/Geometric_mean">geometric mean</a> of <span class="math">\(e^\mathcal{L}\)</span>;</li>
<li><span class="math">\(\alpha=2\)</span> bounds the <a href="https://en.wikipedia.org/wiki/Arithmetic_mean">arithmetic mean</a> of <span class="math">\(e^\mathcal{L}\)</span>;</li>
<li><span class="math">\(\alpha=3\)</span> bounds the <a href="https://en.wikipedia.org/wiki/Root_mean_square">quadratic mean</a> of <span class="math">\(e^\mathcal{L}\)</span>;</li>
<li><span class="math">\(\alpha=4\)</span> bounds the <a href="https://en.wikipedia.org/wiki/Cubic_mean">cubic mean</a> of <span class="math">\(e^\mathcal{L}\)</span>;</li>
<li>and it's also possible to pick <span class="math">\(\alpha=\infty\)</span>, which bounds the maximum value
of <span class="math">\(e^\mathcal{L}\)</span>: it's then equivalent to <span class="math">\(\varepsilon\)</span>-DP.</li>
</ul>
<p>Let's visualize these options using our previous example.</p>
<p><center class="largeimg">
<img alt="Same graph as before, with four red dashed lines labeled alpha=1, 2, 3, and infinity. The infinity line is at 150, corresponding to the maximum of the blue line." src="https://desfontain.es/privacy/images/multiple-averages-plrv.svg">
</center> </p>
<p>Rényi DP, <a href="https://arxiv.org/abs/1702.07476">invented</a> by Ilya Mironov, has a bunch of neat properties.
In particular, it composes nicely, <a href="differential-privacy-awesomeness.html#composition">just like DP</a>. If a mechanism <span class="math">\(A\)</span>
is <span class="math">\((\alpha,\varepsilon_1)\)</span>-Rényi DP and a mechanism <span class="math">\(A'\)</span> is
<span class="math">\((\alpha,\varepsilon_2)\)</span>-DP, then releasing the output of both will be
<span class="math">\((\alpha,\varepsilon_1+\varepsilon_2)\)</span>-DP.</p>
<h1 id="zero-concentrated-differential-privacy">Zero-concentrated differential privacy</h1>
<p>So Rényi DP is pretty neat, but it involves an additional parameter <span class="math">\(\alpha\)</span>.
That's a bit annoying. Choosing <span class="math">\(\varepsilon\)</span> was already difficult. Having to
make a new decision about "how to average the risk" seems even harder. Yet, this
idea of averaging the privacy loss is pretty natural. Ideally, we would like to
keep this intuition, but have a single parameter.</p>
<p>What if we covered all possible values of <span class="math">\(\alpha\)</span> at once? Larger <span class="math">\(\alpha\)</span>
values put more weight on bad events: the "average" also gets larger as <span class="math">\(\alpha\)</span>
grows. So what if we put a bound on the average… but have this bound grow with
<span class="math">\(\alpha\)</span>? This seems like a good idea. But now, the question becomes: how <em>fast</em>
should it grow? There are a lot of increasing functions. But a logarithm doesn't
exactly behave like an exponential!</p>
<p>Since we have a choice, we can think of what other things we'd like from a
single-parameter definition. We saw that <a href="gaussian-noise.html">Gaussian noise</a> was a neat tool
to design DP mechanisms: it would be nice to describe its privacy guarantee in a
simple way with our new definition. Composition is also important, and if
possible, a <em>simple</em> composition result would be better.</p>
<p>To sum up, we're looking for a formulation that:</p>
<ol>
<li>has a single parameter,</li>
<li>corresponds to a larger <span class="math">\(\varepsilon\)</span> for growing values of <span class="math">\(\alpha\)</span>,</li>
<li>describes the guarantee of Gaussian noise in a simple & precise way,</li>
<li>and has a simple composition guarantee.</li>
</ol>
<p>That's exactly what zero-concentrated differential privacy (zCDP) provides.
<a href="https://arxiv.org/abs/1605.02065">Introduced</a> by Mark Bun & Thomas Steinke, it can be interpreted in
simple terms: given a single parameter <span class="math">\(\rho\)</span>, the <span class="math">\(\varepsilon\)</span> corresponding
to each <span class="math">\(\alpha\)</span> must be at most <span class="math">\(\rho\alpha\)</span>. In the formalism above, the
mechanism is <span class="math">\(\rho\)</span>-zCDP if:</p>
<div class="math">$$
\underset{O\sim A(D_1)}{\mathbb{E}} \left[\left(e^{\mathcal{L}_{D_1,D_2}(O)}\right)^{(\alpha-1)}\right] \le \left(e^{\rho\alpha}\right)^{(\alpha-1)}.
$$</div>
<p>It's easy to verify that it matches all the requirements above.</p>
<ol>
<li>The single parameter <span class="math">\(\rho\)</span> corresponds to the arithmetic average of the
privacy loss. (Or, equivalently, to the geometric average of the
<span class="math">\(e^\mathcal{L}\)</span>.)</li>
<li>It guarantees that the relationship between <span class="math">\(\alpha\)</span> and <span class="math">\(\varepsilon\)</span> is at
most linear, which is very simple.</li>
<li>It describes the <a href="gaussian-noise.html">Gaussian mechanism</a> beautifully. Suppose that the
statistics you're computing have a <span class="math">\(L^2\)</span> sensitivity of <span class="math">\(\Delta\)</span>. Then,
adding adding Gaussian noise of variance <span class="math">\(\sigma^2\)</span> to the result. Then the
result satisfies <span class="math">\(\rho\)</span>-zCDP, with <span class="math">\(\rho=\frac{\Delta^2}{2\sigma^2}\)</span>. So much
nicer than the <a href="gaussian-noise.html#formula">formula</a> giving the <span class="math">\((\varepsilon,\delta)\)</span>-DP guarantee!</li>
<li>And composition is a breeze. If a mechanism is <span class="math">\(\rho_1\)</span>-zCDP and another is
<span class="math">\(\rho_2\)</span>-zCDP, then publishing the result of both is
<span class="math">\(\left(\rho_1+\rho_2\right)\)</span>-zCDP.</li>
</ol>
<p>The last two points are super useful to analyze <em>multiple</em> Gaussian mechanisms
at once: we can look at them separately, and add their corresponding <span class="math">\(\rho\)</span>
values. This works even if they use very different noise magnitudes. And the
resulting guarantee is much more precise than if we'd done the accounting with
<span class="math">\((\varepsilon,\delta)\)</span>-DP.</p>
<p>These nice properties are why zCDP has been used in practice for some
high-profile use cases, like the 2020 U.S. Census <a href="https://www2.census.gov/programs-surveys/decennial/2020/program-management/data-product-planning/2010-demonstration-data-products/ppmf20210608/2021-06-08-privacy-loss_budgetallocation.pdf">Redistricting Data</a>.
If you want to release a lot of statistics, you might benefit from using this
notion in your privacy analysis as well.</p>
<h1 id="tldr">tl;dr</h1>
<p>Can we describe all the definitions we've seen so far in a tweet-length summary?
Here's an attempt.</p>
<ul>
<li><span class="math">\(\varepsilon\)</span>-DP: the absolute worst case is <span class="math">\(\varepsilon\)</span>.</li>
<li><span class="math">\(\left(\varepsilon,\delta\right)\)</span>-DP: the worst case is <span class="math">\(\varepsilon\)</span>, almost
always.</li>
<li><span class="math">\(\left(\alpha,\varepsilon\right)\)</span>-Rényi DP: the average case is <span class="math">\(\varepsilon\)</span>,
and <span class="math">\(\alpha\)</span> tells you which average function to use.</li>
<li><span class="math">\(\rho\)</span>-zCDP: many <span class="math">\(\left(\alpha,\varepsilon\right)\)</span>-Rényi DP guarantees at
once, well-chosen for convenience.</li>
</ul>
<p>Simple, right?</p>
<hr>
<p><small></p>
<p>Note: I optimized this article for simplicity. I tried to find the simplest
possible intuition for these notions, and made up a neat storyline to introduce
one after the other. This came at a cost in historical accuracy. If your main
goal was getting an intuitive understanding of these definitions, then you can
stop here. If you're also interested in learning about the history of these
notions, keep reading.</p>
<p>Contrary to the story above, zero-concentrated DP was introduced <em>before</em> Rényi
DP. This work itself built on a prior definition, <a href="https://arxiv.org/abs/1603.01887"><em>concentrated DP</em></a>,
invented by Dwork and Rothblum. This prior notion says that if you take the PLRV
and subtract its mean, you get a distribution that's "smaller than a Gaussian".</p>
<p>Concentrated DP was a fruitful notion, used to prove tighter composition
theorems for <span class="math">\((\varepsilon,\delta)\)</span>-DP. It also described the privacy properties
of the Gaussian mechanism in a neater way. But has also some shortcomings: it
was not closed under post-processing, and its formulation was fairly complex.
This is what zero-concentrated DP was introduced to fix: it formalized the "PLRV
is smaller than a Gaussian" intuition in a simpler way, keeping the advantages
without the problems.</p>
<p>The original goal of both notions was to get better composition results, not to
average risk. Rényi DP, introduced afterwards, followed this line of research.
Fixing the parameter <span class="math">\(\alpha\)</span> was a way of getting more flexibility in the
privacy analysis, in particular for machine learning use cases.</p>
<p></small></p>
<hr>
<p><small>Thanks to Anthony Caruso for letting me know about a mistake in a
previous version of this post.</small></p>
<div class="footnote">
<hr>
<ol>
<li id="fn:partial">
<p><a href="https://differentialprivacy.org/average-case-dp/">This blog post</a>
gives a few examples of what this can look like, and the dangers of doing so. <a class="footnote-backref" href="#fnref:partial" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>A list of real-world uses of differential privacy2021-10-01T00:00:00+02:002023-09-22T00:00:00+02:00Damien Desfontainestag:desfontain.es,2021-10-01:/privacy/real-world-differential-privacy.html<p>A list of practical deployments of differential privacy, along with their privacy parameters.</p><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p></small></p>
<hr>
<p><span class='lettrine'>T</span><strong>his</strong> article is a list of real-world
deployments of differential privacy, along with their privacy parameters. One
day, we might have a proper <a href="https://journalprivacyconfidentiality.org/index.php/jpc/article/view/689">Epsilon Registry</a>, but in the meantime…</p>
<p>First, some notes.</p>
<ul>
<li>The main list only includes projects with a publicly documented value of the
privacy parameters, including about what the <a href="why-not-differential-privacy.html#privacy-units">privacy unit</a> is. Projects
that don't publish this information, but mention using DP, are listed at the
end.</li>
<li>All use cases use <a href="local-global-differential-privacy.html#central">central DP</a> unless specified otherwise.</li>
<li>The list is sorted by alphabetical order of the organization publishing the
data.</li>
<li>When a project uses open-source differential privacy tooling, I added a link
to it.</li>
<li>I also added some caveats and general comments at the end of this post.</li>
</ul>
<p>If you'd like to add or correct something, please let me know! My contact info
is at the bottom of this page.</p>
<h1 id="apple">Apple</h1>
<p><center>
<img alt="An architecture diagram taken from Apple's differential privacy paper" src="https://desfontain.es/privacy/images/apple-dp-diagram.png">
</center></p>
<p>Apple uses <a href="local-global-differential-privacy.html#local">local DP</a> to collect some data from end-user devices running
iOS or macOS. The process is documented in a <a href="https://www.apple.com/privacy/docs/Differential_Privacy_Overview.pdf">high-level overview
document</a> and a <a href="https://docs-assets.developer.apple.com/ml-research/papers/learning-with-privacy-at-scale.pdf">detailed paper</a>. All use
<span class="math">\(\varepsilon\)</span>-DP, the values of the privacy parameter are described below, with
a privacy unit of user-day.</p>
<ul>
<li><em>QuickType suggestions</em> learns previously-unknown words typed by sufficiently
many users, using <span class="math">\(\varepsilon=16\)</span>.</li>
<li><em>Emoji suggestions</em> calculates which emojis are most popular among users,
using <span class="math">\(\varepsilon=4\)</span>.</li>
<li><em>Lookup hints</em> collects data on actions taken from iOS <a href="https://support.apple.com/guide/iphone/search-with-iphone-iph3c511548/ios">Search
suggestions</a>. (I think. It's not very explicit.) It uses
<span class="math">\(\varepsilon=8\)</span>.</li>
<li><em>Health Type Usage</em> estimates which health types are most used in the
HealthKit app, using <span class="math">\(\varepsilon=2\)</span>.</li>
<li><em>Safari Energy Draining Domains</em> and <em>Safari Crashing Domains</em> collect data on
web domains: which domains are most likely to cause high energy consumption or
crashes, respectively. Both features use a common budget of <span class="math">\(\varepsilon=8\)</span>.</li>
<li><em>Safari Autoplay Intent Detection</em> collects data about websites that auto-play
videos with sound: in which of these domains are users most likely to mute vs.
keep playing the video? It uses <span class="math">\(\varepsilon=16\)</span>.</li>
</ul>
<p><button class="toggleDetails"></button></p>
<div style="display: none; border-left: double; padding-left: 10px">
<p>The documented privacy unit is each data collection event. The devices send a
limited number of such events per day: I translated all guarantees to use a
privacy unit of user-day. Apple also does some de-identification and <a href="local-global-differential-privacy.html#shuffling">shuffling</a>
(see in Section 3.2.2 of <a href="https://docs-assets.developer.apple.com/ml-research/papers/learning-with-privacy-at-scale.pdf">their paper</a>). Taking this into account
would presumably lead to tighter central DP guarantees.</p>
</div>
<h1 id="facebook">Facebook</h1>
<h4 id="full-urls-data-set">Full URLs Data Set</h4>
<p>The <a href="https://dataverse.harvard.edu/file.xhtml?persistentId=doi:10.7910/DVN/TDOAPG/DGSAMS&version=6.2"><em>Full URLs Data Set</em></a> provides data on user interactions with web
pages shared on Facebok. The privacy unit is each individual action: this can be
e.g. "Alice shared URL foo.com", or "Bob viewed a post containing URL bar.org".
For each type of action, the privacy parameter is chosen to protect 99% of users
with <span class="math">\((\varepsilon,\delta)\)</span>-DP, for <span class="math">\(\varepsilon=0.45\)</span> and <span class="math">\(\delta=10^{-5}\)</span>.
Across all metrics, 96.6% of users are protected with <span class="math">\((\varepsilon,\delta)\)</span>-DP
with <span class="math">\(\varepsilon=1.453\)</span> and <span class="math">\(\delta=10^{-5}\)</span>.
<button class="toggleDetails"></button></p>
<div style="display: none; border-left: double; padding-left: 10px">
<p>Behind the scenes, this uses <span class="math">\(\rho\)</span>-<a href="renyi-dp-zero-concentrated-dp.html">zero-concentrated DP</a>, with
<span class="math">\(\rho=0.0052\)</span> for 99% users for each action type, and an overall <span class="math">\(\rho=0.0728\)</span>
for 96.6% of users. The paper refers to two additional DP operations:</p>
<ul>
<li>URLs that have not been shared by enough users (according to a DP count) are
discarded;</li>
<li>the algorithm also calculates the 99% percentile of each action in a DP way.</li>
</ul>
<p>It does not quantify the privacy budget used for these two operations.</p>
</div>
<h4 id="movement-range-maps">Movement Range Maps</h4>
<p><center>
<img alt="An animated map of the "Stay Put" metric in Facebook's Movement Range Maps" src="https://desfontain.es/privacy/images/stay-put-map-fb.gif">
</center></p>
<p>The <a href="https://research.fb.com/blog/2020/06/protecting-privacy-in-facebook-mobility-data-during-the-covid-19-response/"><em>Movement Range Maps</em></a> quantify the changes in mobility of
Facebook users during the COVID-19 pandemic. There are two metrics: how much
their users move during each day, and how many people are generally staying at
home. Each metric uses a daily value <span class="math">\(\varepsilon=1\)</span>, so the overall privacy
budget is <span class="math">\(\varepsilon=2\)</span> with user-day as a privacy unit.
<button class="toggleDetails"></button></p>
<div style="display: none; border-left: double; padding-left: 10px">
<p>The <a href="https://research.fb.com/blog/2020/06/protecting-privacy-in-facebook-mobility-data-during-the-covid-19-response/">blog post</a> also mentions that regions with fewer than 300 users
are omitted. This process doesn't appear to be done in a DP way.</p>
</div>
<h1 id="google">Google</h1>
<h4 id="community-mobility-reports">Community Mobility Reports</h4>
<p><center>
<img alt="Two graphs comparing time spent in residential vs. workspace places compared to a baseline" src="https://desfontain.es/privacy/images/community-mobility-reports.png">
</center></p>
<p>The <a href="https://arxiv.org/abs/2004.04145"><em>Community Mobility Reports</em></a> quantify changes in mobility patterns
during the COVID-19 pandemic: how many people went to their workplace or to
specific kinds of public places, and how long people spent at home. Each metric
uses <span class="math">\(\varepsilon=0.44\)</span> per day, and each user contributes to at most six
metrics per day. Thus, the total privacy budget is <span class="math">\(\varepsilon=2.64\)</span>, with
user-day as a privacy unit. The data was made differentially private using
<a href="https://github.com/google/differential-privacy">GoogleDP</a><sup id="fnref:googledp"><a class="footnote-ref" href="#fn:googledp">1</a></sup>.
<button class="toggleDetails"></button></p>
<div style="display: none; border-left: double; padding-left: 10px">
<p>The <a href="https://arxiv.org/abs/2004.04145">paper</a> also mentions using more privacy budget used to update the way
the metrics are computed. This additional budget isn't quantified exactly.</p>
</div>
<h4 id="next-word-prediction-model-on-gboard">Next-word prediction model on Gboard</h4>
<p><center>
<img alt="A diagram showing model updates in parameter space, comparing DP-FTRL with DP-SGD and true gradients." src="https://desfontain.es/privacy/images/dp-ftrl.png">
</center></p>
<p>Google uses <a href="https://en.wikipedia.org/wiki/Federated_learning">federated learning</a> along with DP to build the <a href="https://ai.googleblog.com/2022/02/federated-learning-with-formal.html">next-word
prediction model</a> for the Spanish-language version of Gboard, a
virtual keyboard application for Android. It uses <span class="math">\((\varepsilon,\delta)\)</span>-DP with
<span class="math">\(\varepsilon=6.92\)</span> and <span class="math">\(\delta=10^{-5}\)</span>, with a user as a privacy unit. The
model was trained using <a href="https://www.tensorflow.org/federated">TensorFlow Federated</a> and <a href="https://www.tensorflow.org/responsible_ai/privacy/guide">TensorFlow
Privacy</a>.
<button class="toggleDetails"></button></p>
<div style="display: none; border-left: double; padding-left: 10px">
<p>The privacy guarantees are reported using <span class="math">\(\rho\)</span>-<a href="renyi-dp-zero-concentrated-dp.html">zero-concentrated DP</a>,
with a budget of <span class="math">\(\rho=0.81\)</span>.</p>
</div>
<h4 id="search-trends-symptoms-dataset">Search Trends Symptoms Dataset</h4>
<p><center>
<img alt="An animated visualization of searches for Fever in the US through 2020, using Google's Search Trends Symptoms Dataset" src="https://desfontain.es/privacy/images/google-sst-map.gif">
</center></p>
<p>The <a href="https://arxiv.org/abs/2009.01265"><em>Search Trends Symptoms Dataset</em></a> measures the volume of Google
searches related to a variety of symptoms. It uses <span class="math">\(\varepsilon=1.68\)</span>, with a
user-day privacy unit; the release was generated using <a href="https://github.com/google/differential-privacy">GoogleDP</a>.</p>
<h4 id="urban-mobility-data">Urban mobility data</h4>
<p><center>
<img alt="Figure 1 from the paper linked below, showing visualizations of mobility and location hotspots in 7 large cities" src="https://desfontain.es/privacy/images/urban-mobility-data.webp">
</center></p>
<p>Google <a href="https://ai.googleblog.com/2019/11/new-insights-into-human-mobility-with.html">shared mobility data</a> with researchers, using DP to
anonymize it. The resulting <a href="https://www.nature.com/articles/s41467-019-12809-y">paper</a> says that this data sharing
scheme used <span class="math">\((\varepsilon,\delta)\)</span>-DP with <span class="math">\(\varepsilon=0.66\)</span> and
<span class="math">\(\delta=2.1\cdot10^{-29}\)</span>. The privacy unit is whether a given user made a trip
from one location to another location during one week; both locations being
fixed areas of size <span class="math">\(\approx1.3\)</span>km².
<button class="toggleDetails"></button></p>
<div style="display: none; border-left: double; padding-left: 10px">
<p>The privacy unit was not explicitly given in the original paper, which could
have given the impression that the data release used a user-level privacy unit.
Other researchers <a href="https://www.nature.com/articles/s41467-021-27566-0">pointed this out</a>, after which the original
authors published a <a href="https://www.nature.com/articles/s41467-021-27567-z">clarification</a> making the privacy
guarantees more explicit.</p>
</div>
<h4 id="vaccination-search-insights">Vaccination Search Insights</h4>
<p>The <a href="https://arxiv.org/abs/2107.01179"><em>Vaccination Search Insights</em></a> quantify trends in Google searches
related to COVID-19 vaccination. It uses <span class="math">\((\varepsilon,\delta)\)</span>-DP with
<span class="math">\(\varepsilon=2.19\)</span> and <span class="math">\(\delta=10^{-5}\)</span>, with user-day as a privacy unit; the
data was generated using <a href="https://github.com/google/differential-privacy">GoogleDP</a>.
<button class="toggleDetails"></button></p>
<div style="display: none; border-left: double; padding-left: 10px">
<p>A later <a href="https://blog.research.google/2023/05/differentially-private-clustering-for.html">blog post</a> suggests that the query classification mentioned as
pre-processing in the <a href="https://arxiv.org/abs/2107.01179">technical paper</a> is also done in a DP way. The
privacy budget involved in this step is not public, and is not counted towards
the reported guarantees. The code for this step is also part of
<a href="https://github.com/google/differential-privacy">GoogleDP</a>.</p>
</div>
<h1 id="linkedin">LinkedIn</h1>
<h4 id="audience-engagements-api">Audience Engagements API</h4>
<p><center>
<img alt="An architecture diagram from LinkedIn's Audience Engagements API paper" src="https://desfontain.es/privacy/images/linkedin-dp-api-diagram.png">
</center></p>
<p>The <a href="https://arxiv.org/abs/2002.05839"><em>Audience Engagements API</em></a> is the only interactive query system in
this list. It allows marketers to get information about LinkedIn users engaging
with their content. Each query returns <span class="math">\((\varepsilon,\delta)\)</span>-DP with
<span class="math">\(\varepsilon=0.15\)</span> and <span class="math">\(\delta=10^{-10}\)</span>, with a user as a privacy unit. Each
analyst can send multiple queries, but a monthly cap limits how many: the total
<span class="math">\((\varepsilon,\delta)\)</span> budget is <span class="math">\(\varepsilon=34.9\)</span> and <span class="math">\(\delta=7\cdot10^{-9}\)</span>,
with a privacy unit of user-month-analyst.
<button class="toggleDetails"></button></p>
<div style="display: none; border-left: double; padding-left: 10px">
<p>The system also implements additional measures to prevent averaging attacks: new
data is loaded daily, and seeded noise is used so the same query on the same day
will always return the same answer.</p>
</div>
<h4 id="labor-market-insights">Labor Market Insights</h4>
<p>The <a href="https://arxiv.org/abs/2010.13981"><em>Labor Market Insights</em></a> measure trends in people changing their
occupation on LinkedIn. There are three types of reports.</p>
<ul>
<li><em>Who is hiring?</em> lists the companies who are hiring most. It uses
<span class="math">\((\varepsilon,\delta)\)</span>-DP to protect each hiring event (a LinkedIn user
changing their occupation), with <span class="math">\(\varepsilon=14.4\)</span> and
<span class="math">\(\delta=1.2\cdot10^{-9}\)</span>.</li>
<li><em>What jobs are available?</em> enumerates the job titles that most people are
being hired for. It also uses <span class="math">\((\varepsilon,\delta)\)</span>-DP to protect each hiring
event, with <span class="math">\(\varepsilon=14.4\)</span> and <span class="math">\(\delta=1.2\cdot10^{-9}\)</span>.</li>
<li><em>What skills are needed?</em> lists the most popular skills for the jobs above. It
protects each LinkedIn user's skills information during a single month with
<span class="math">\(\varepsilon=0.3\)</span> and <span class="math">\(\delta=3\cdot10^{-10}\)</span>.</li>
</ul>
<p>This suggests a total <span class="math">\(\varepsilon=28.8\)</span> and <span class="math">\(\delta=2.4\cdot10^{-9}\)</span>-DP for
hiring events, and <span class="math">\(\varepsilon=0.3\)</span> and <span class="math">\(\delta=3\cdot10^{-10}\)</span> for skill
information during a single month. However, there are many subtleties involved
in the above analysis. It's very possible to interpret the paper differently.
<button class="toggleDetails"></button></p>
<div style="display: none; border-left: double; padding-left: 10px">
<ol>
<li>The privacy parameters listed in the <a href="https://arxiv.org/abs/2010.13981">paper</a> are three times smaller.
However, each report covers 3 months of data, and reports are published
monthly: a single hiring event will appear in three distinct reports.</li>
<li>For <em>What skills are needed?</em>, each monthly report looks back at 5 years of
data. So if skill data for a user doesn't change during a 5-year period, the
total budget eventually reaches <span class="math">\(\varepsilon=6\)</span> and <span class="math">\(\delta=6\cdot10^{-9}\)</span>.</li>
<li>Adding the <span class="math">\(\varepsilon\)</span> and <span class="math">\(\delta\)</span> values together, like I did, is simple,
but only give loose bounds on the overall privacy budget. We can probably
find tighter bounds using advanced composition theorems or other methods for
privacy accounting.</li>
<li>The paper also indicates that 95% of people in the dataset have at most one
hiring event in a 3-month period.</li>
<li>The <em>What skills are needed?</em> report also uses a non-DP pre-processing step.
This makes it technically impossible to provide an exact DP guarantee.</li>
</ol>
</div>
<h1 id="microsoft">Microsoft</h1>
<h4 id="global-victim-perpetrator-synthetic-dataset">Global victim-perpetrator synthetic dataset</h4>
<p>Microsoft collaborated with the <a href="https://www.iom.int/">International Organization for Migration</a>
to publish the <a href="https://www.ctdatacollaborative.org/global-victim-perpetrator-synthetic-dataset"><em>Global Victim-Perpetrator Synthetic Dataset</em></a>, which
provides information about victims and perpetrators of trafficking. The release
uses <span class="math">\((\varepsilon,\delta)\)</span>-DP with <span class="math">\(\varepsilon=12\)</span> and
<span class="math">\(\delta=5.8\cdot10^{-6}\)</span>; the privacy unit is a victim in the original dataset.
It uses <a href="https://github.com/microsoft/synthetic-data-showcase">custom code</a> to generate the data.</p>
<h4 id="telemetry-collection-in-windows">Telemetry collection in Windows</h4>
<p>Microsoft collects <a href="https://www.microsoft.com/en-us/research/publication/collecting-telemetry-data-privately/">telemetry data in Windows</a>. The process used to
get information about how much time users spend using particular apps uses
<a href="local-global-differential-privacy.html#local">local DP</a>, with <span class="math">\(\varepsilon=1.672\)</span>, and a privacy unit of user-6-hours.</p>
<h4 id="us-broadband-coverage-dataset">U.S. Broadband Coverage Dataset</h4>
<p><center>
<img alt="A map of the US where each postal code is colored according to the fraction of devices using broadband" src="https://desfontain.es/privacy/images/broadband-coverage.png">
</center></p>
<p>The <a href="https://arxiv.org/abs/2103.14035"><em>U.S. Broadband Coverage Dataset</em></a> quantifies the percentage of users
having access to high-speed Internet across the US. It uses <span class="math">\(\varepsilon\)</span>-DP
with <span class="math">\(\varepsilon=0.2\)</span>, the privacy unit is a user. The data was privatized
using <a href="https://smartnoise.org/">OpenDP SmartNoise</a>.</p>
<h1 id="ohmconnect">OhmConnect</h1>
<p><center>
<img alt="A screenshot of a UI visualizing the impact of DP on queries returning average possible energy savings among a group" src="https://desfontain.es/privacy/images/energy-dp.png">
</center></p>
<p>The <a href="https://edp.recurve.com/"><em>Energy Differential Privacy</em></a> project enables sharing of smart meter
data. In one <a href="https://assets.website-files.com/5cb0a177570549b5f11b9550/5ffddb83b5ea5d67f5c43661_Quantifying%20The%20OhmConnect%20Virtual%20Power%20Plant%20During%20the%20California%20Blackouts.pdf">project</a>, <a href="https://www.recurve.com/">Recurve</a> helped <a href="https://www.ohmconnect.com">OhmConnect</a> share data from
their virtual power plant. This project uses <span class="math">\((\varepsilon,\delta)\)</span>-DP with
<span class="math">\(\varepsilon=4.72\)</span> and <span class="math">\(\delta=5.06\cdot10^{-9}\)</span>, with user as a privacy unit.
The project uses both <a href="https://github.com/recurve-inc/eeprivacy">custom open-source code</a> and Google's
<a href="https://github.com/google/differential-privacy">open-source DP libraries</a>. <button class="toggleDetails"></button></p>
<div style="display: none; border-left: double; padding-left:
10px">
<p>The privacy parameters appearing in the <a href="https://assets.website-files.com/5cb0a177570549b5f11b9550/5ffddb83b5ea5d67f5c43661_Quantifying%20The%20OhmConnect%20Virtual%20Power%20Plant%20During%20the%20California%20Blackouts.pdf">technical paper</a> are
different. The accounting uses amplification by sampling, with a sampling factor
of <span class="math">\(\eta=0.124\)</span>. However, the paper converts a pre-amplification
<span class="math">\(\varepsilon_{orig}=6.8\)</span> into <span class="math">\(\varepsilon=\eta\cdot\varepsilon_{orig}=0.843\)</span>.
The correct formula is
<span class="math">\(\varepsilon=\log\left(1+\mu\left(e^{\varepsilon_{orig}}-1\right)\right)\)</span> (see
Theorem 9 in <a href="https://arxiv.org/abs/1807.01647">summary of results</a>), which gives <span class="math">\(\varepsilon=4.72\)</span>.
The <span class="math">\(\delta\)</span> listed above is also amplified (with <span class="math">\(\delta=\mu\delta_{orig}\)</span>),
the one reported in the paper is not.</p>
<p>Note that the amplification result assumes uniformly random sampling with
replacement. But the paper also mentions a stratified sampling methodology,
which is slightly different: it's unclear whether the amplification result still
applies. If not, then the privacy parameters are <span class="math">\(\varepsilon=6.8\)</span> and
<span class="math">\(\delta=4.08\cdot10^{-8}\)</span>.</p>
</div>
<h1 id="united-states-census-bureau">United States Census Bureau</h1>
<h4 id="post-secondary-employment-outcomes">Post-Secondary Employment Outcomes</h4>
<p>The <a href="https://lehd.ces.census.gov/data/pseo_experimental.html"><em>Post-Secondary Employment Outcomes</em></a> provide data about the earning
and employment of college graduates. The <a href="https://lehd.ces.census.gov/doc/PSEOTechnicalDocumentation.pdf">technical documentation</a>
mentions two statistics using <span class="math">\(\varepsilon\)</span>-DP with <span class="math">\(\varepsilon=1.5\)</span>, for a
total privacy budget of <span class="math">\(\varepsilon=3\)</span>. The privacy unit is a person in the
dataset, and the methods are described in detail in <a href="https://journalprivacyconfidentiality.org/index.php/jpc/article/view/722">this paper</a>.</p>
<h4 id="2020-census-redistricting-data">2020 Census Redistricting Data</h4>
<p><center>
<img alt="A screenshot from the 2020 Census Demographic Data Map Viewer" src="https://desfontain.es/privacy/images/census-map-viewer.png">
</center></p>
<p>The <a href="https://www.census.gov/newsroom/press-releases/2021/2020-census-key-parameters.html"><em>2020 Census Redistricting Data</em></a> contain demographic
information about the U.S. population. It is protected with
<span class="math">\((\varepsilon,\delta)\)</span>-DP with <span class="math">\(\varepsilon=13.64\)</span> and <span class="math">\(\delta=10^{-5}\)</span>, where
the privacy unit is a person in the dataset. This uses custom code that was
<a href="https://github.com/uscensusbureau/DAS_2020_Redistricting_Production_Code">published on GitHub</a>.
<button class="toggleDetails"></button></p>
<div style="display: none; border-left: double; padding-left: 10px">
<p>The <a href="https://www.census.gov/programs-surveys/decennial-census/decade/2020/planning-management/process/disclosure-avoidance/protections-by-data-product.html#redistricting-data">privacy accounting</a> is done with
<span class="math">\(\rho\)</span>-<a href="renyi-dp-zero-concentrated-dp.html">zero-concentrated DP</a>, with a global budget of <span class="math">\(\rho=2.63\)</span>; the
official document reports privacy parameters of <span class="math">\(\varepsilon=19.61\)</span> and
<span class="math">\(\delta=10^{-10}\)</span>. </p>
</div>
<h4 id="2020-census-demographic-and-housing-characteristics-file">2020 Census Demographic and Housing Characteristics File</h4>
<p>As part of the 2020 Decennial Census, the U.S. Census is publishing the
<em>Demographic and Housing Characteristics</em> File (DHC), which provides data about
demographic and households characteristics of the U.S. population. The first
part of this release, the <a href="https://www.census.gov/newsroom/press-releases/2023/2020-census-detailed-dhc-file-a.html"><em>Detailed DHC-A</em></a>, was published with
<span class="math">\((\varepsilon,\delta)\)</span>-DP with <span class="math">\(\varepsilon=49.95\)</span> and <span class="math">\(\delta=10^{-5}\)</span>, using
<a href="https://tmlt.dev">Tumult Analytics</a>.
<button class="toggleDetails"></button></p>
<div style="display: none; border-left: double; padding-left: 10px">
<p>The <a href="https://www2.census.gov/programs-surveys/decennial/2020/technical-documentation/complete-tech-docs/detailed-demographic-and-housing-characteristics-file-a/2020census-detailed-dhc-a-techdoc.pdf#page=60">privacy accounting</a> is done with
<span class="math">\(\rho\)</span>-<a href="renyi-dp-zero-concentrated-dp.html">zero-concentrated DP</a>, with a global budget of <span class="math">\(\rho=19.776\)</span>. </p>
</div>
<h1 id="wikimedia-foundation">Wikimedia Foundation</h1>
<p>The <a href="https://wikimediafoundation.org/">Wikimedia Foundation</a>, helped by <a href="https://tmlt.io">Tumult Labs</a>, published
statistics about how many distinct users visited each Wikipedia page on each
day, from each country. The <a href="https://diff.wikimedia.org/2023/06/21/new-dataset-uncovers-wikipedia-browsing-habits-while-protecting-users/">data publication</a> also covers other
<a href="https://meta.wikimedia.org/wiki/Our_projects">Wikimedia projects</a>, and is split in three parts.</p>
<ul>
<li>Data from July 1st, 2015 to February 8th, 2017 is protected with
<span class="math">\(\varepsilon\)</span>-DP with <span class="math">\(\varepsilon=1\)</span>, the privacy unit being 300 page views
per day.</li>
<li>Data from February 9th, 2017 to February 5th, 2023 is protected with
<span class="math">\(\varepsilon\)</span>-DP with <span class="math">\(\varepsilon=1\)</span>, the privacy unit being 30 page views
per day.</li>
<li>Data from February 6th, 2023 onwards is protected with
<span class="math">\((\varepsilon,\delta)\)</span>-DP with <span class="math">\(\varepsilon=0.85\)</span> and <span class="math">\(\delta=10^{-5}\)</span>, with a
user-day privacy unit.</li>
</ul>
<p>A <a href="https://arxiv.org/abs/2308.16298">technical paper</a> explains the why different units of privacy and
privacy budgets are used for different periods.
<button class="toggleDetails"></button></p>
<div style="display: none; border-left: double; padding-left: 10px">
<p>For the most recent data, the privacy accounting is done with
<span class="math">\(\rho\)</span>-<a href="renyi-dp-zero-concentrated-dp.html">zero-concentrated DP</a> with a budget of <span class="math">\(\rho=0.015\)</span>, which also
corresponds to <span class="math">\(\varepsilon=1\)</span> for <span class="math">\(\delta=10^{-7}\)</span>. </p>
</div>
<h1 id="other-deployments">Other deployments</h1>
<p>This list is almost certainly incomplete. Again, don't hesitate to reach out if
you'd like me to add or correct something!</p>
<ul>
<li>Apple and Google's <a href="https://covid19.apple.com/contacttracing">Exposure Notification framework</a> has an
<a href="https://covid19-static.cdn-apple.com/applications/covid19/current/static/contact-tracing/pdf/ENPA_White_Paper.pdf">analytics</a> component that uses <a href="local-global-differential-privacy.html#shuffling">shuffled DP</a>. The
paper mentions a local <span class="math">\(\varepsilon=8\)</span> and corresponding central values of
<span class="math">\(\varepsilon\)</span> depending on how many users participate and on the central
<span class="math">\(\delta\)</span> chosen. However, it does not specify the privacy unit, the number of
aggregations, nor the minimal number of participating users.</li>
<li>Google mentions using DP in two <a href="https://developers.googleblog.com/2019/09/enabling-developers-and-organizations.html">Google Maps features</a>: the first
quantifies how busy public places are during the day, the second which
restaurant's dishes are most popular. It does not specify the privacy
parameters used nor the exact method used to generate the data.</li>
<li>Google's <a href="https://security.googleblog.com/2014/10/learning-statistics-with-privacy-aided.html">RAPPOR</a> used to collect browsing information in Google Chrome with
<a href="local-global-differential-privacy.html#local">local DP</a>. It is now <a href="https://github.com/chromium/chromium/blob/72ceeed2ebcd505b8d8205ed7354e862b871995e/chrome/browser/prefs/browser_prefs.cc#L509">deprecated</a>.</li>
<li>Google mentions using DP and federated learning to train models to improve
<a href="https://ai.googleblog.com/2023/03/distributed-differential-privacy-for.html">text selection and copying</a> on Android. The deployment uses
distributed DP, which provides similar guarantees to <a href="local-global-differential-privacy.html#local">local DP</a>, with
additional assumptions about the adversary (which must be honest-but-curious).
The value of <span class="math">\(\varepsilon\)</span> is reported to be "in the hundreds", but not
precisely specified; the privacy unit is also not reported.</li>
<li>The <a href="https://www.irs.gov/">Internal Revenue Service</a> and the <a href="https://www.ed.gov/">U.S. Department of
Education</a>, helped by <a href="https://tmlt.io">Tumult Labs</a>, used DP to publish college
graduate income summaries. The data was generated using <a href="https://tmlt.dev">Tumult
Analytics</a> and published on the <a href="https://collegescorecard.ed.gov/">College Scorecard</a>
website. The project is outlined in <a href="https://www.tmlt.io/research/how-tumult-labs-helped-irs-support-educational-accountability-with-differential-privacy">this post</a>, but no specific
privacy parameters are given.</li>
<li>Microsoft's <a href="https://www.microsoft.com/en-us/research/group/msai/articles/assistive-ai-makes-replying-easier-2/">Assistive AI</a> automatically suggests replies to messages in
Office tools. It provides <span class="math">\((\varepsilon,\delta)\)</span>-DP with <span class="math">\(\varepsilon=4\)</span> and
<span class="math">\(\delta<10^{-7}\)</span>, but does not specify what the privacy unit is.<ul>
<li>A separate <a href="https://www.microsoft.com/en-us/research/blog/privacy-preserving-machine-learning-maintaining-confidentiality-and-preserving-trust/">blog post</a> by Microsoft suggests that this choice of
<span class="math">\(\varepsilon=4\)</span> is a policy standard across use cases for differentially
private machine learning, and applies to the data of each user over a
period of 6 months.</li>
</ul>
</li>
<li>Microsoft also mentions using DP in <a href="https://download.microsoft.com/download/D/1/F/D1F0DFF5-8BA9-4BDF-8924-7816932F6825/Differential_Privacy_for_Everyone.pdf">Workplace Analytics</a>: this allows
managers to see data about their team's interactions with workplace tools. No
specific information about privacy parameters is given.</li>
<li><a href="https://spectus.ai/">Spectus</a> published a <a href="https://spectus.ai/social-impact/evacuation-dashboard/">dashboard</a> containing DP metrics about
mobility trends during Hurricane <a href="https://en.wikipedia.org/wiki/Hurricane_Irma">Irma</a>, and the page suggests that they
generated similar datasets for other natural disasters. The
<a href="https://spectus.ai/wp-content/uploads/2022/10/Spectus_DPWhitepaper_v01b.pdf">whitepaper</a> mentions that <a href="https://smartnoise.org/">OpenDP SmartNoise</a> was
used to generate four <span class="math">\(\varepsilon\)</span>-DP metrics for a total <span class="math">\(\varepsilon=10\)</span>;
the privacy unit is not specified.</li>
<li>The U.S. Census Bureau published <a href="https://lehd.ces.census.gov/applications/help/onthemap.html#!what_is_onthemap">OnTheMap</a> in 2008: this was the
first-ever real-world deployment of DP. It provides statistics on where US
workers are employed and where they live. The DP process is described in
<a href="https://lehd.ces.census.gov/doc/help/ICDE08_conference_0768.pdf">paper</a>, but I haven't found the privacy parameters published
anywhere.</li>
<li>The U.S. Census Bureau published <a href="https://www.census.gov/topics/business-economy/disclosure/data/tables/cbp-privacy-demonstration-tables.html">demonstration tables</a> for their
County Business Patterns data product, providing information about business
establishments in the US. The <a href="https://www.census.gov/topics/business-economy/disclosure/about.html">About page</a> mentions the use of
<span class="math">\(\rho\)</span>-<a href="renyi-dp-zero-concentrated-dp.html">zero-concentrated DP</a> with <span class="math">\(\rho=12.058\)</span>, which gives
<span class="math">\(\varepsilon=35.62\)</span> for <span class="math">\(\delta=10^{-5}\)</span>, but does not specify the privacy
unit.</li>
</ul>
<p>There are (many) other examples of companies and organizations saying they use
DP. I only added them here if they point to a specific project or feature.</p>
<p>Finally, many scientific papers report experimental results on real datasets.
Most don't mention whether the system was deployed. I did not attempt to list
those.</p>
<h1 id="caveats-comments">Caveats & comments</h1>
<h4 id="whats-a-user">What's a user?</h4>
<p>Most of these projects have <em>user</em> as part of their privacy unit. This can mean
slightly different things depending on the project: a device (for telemetry
collection), an account (for online services), a household (for smart meter
data), and so on. This means that an individual who uses multiple devices or
accounts on the same online service might get weaker privacy guarantees. This
subtlety is not always made explicit.</p>
<h4 id="replacement-vs-additionremoval">Replacement vs. addition/removal</h4>
<p>In differential privacy, the definition of the two neighboring datasets can be
of two types. Do you <em>change</em> the data of one person? Or do you <em>add</em> or
<em>remove</em> a user? This subtlety is also not always explicit, and I've ignored it
in the list above.</p>
<h4 id="zero-concentrated-differential-privacy">Zero-concentrated differential privacy</h4>
<p>Multiple data releases use <a href="renyi-dp-zero-concentrated-dp.html">zero-concentrated DP</a> to do the privacy budget
accounting. Some report guarantees using this definition, others convert the
guarantees to <span class="math">\((\varepsilon, \delta)\)</span>-DP in communication materials. To make the
comparison easier, I converted all these guarantees to <span class="math">\((\varepsilon,\delta)\)</span>-DP
with <span class="math">\(\delta=10^{-5}\)</span>; even when the reported <span class="math">\(\delta\)</span> is different.
<button class="toggleDetails"></button></p>
<div style="display: none; border-left: double; padding-left: 10px">
<p>The conversion was done using Proposition 1.3 in <a href="https://arxiv.org/abs/1605.02065">this paper</a>:
<span class="math">\(\varepsilon=\rho+2\sqrt{\rho\ln\left(1/\delta\right)}\)</span>. There are tighter (and
significantly more complex) conversion results available in the literature, so
the actual <span class="math">\(\varepsilon\)</span> will be slightly smaller than reported here.</p>
</div>
<h4 id="comparing-projects">Comparing projects</h4>
<p>You should not use this list to make broad statements or comparisons about the
privacy posture of different organizations. Differential privacy parameters are
a very small part of the story, even for these specific projects. How was the
data collected? How long is it kept? How sensitive is it? Who has access to the
input and output data? Answering these questions is crucial to put each DP
deployment and its parameters in context. </p>
<p>In addition, different privacy units also make simple comparisons fairly
meaningless. Even across time periods, the semantics are subtle. As an example,
consider two DP processes.</p>
<ul>
<li>Process <span class="math">\(A\)</span> uses a privacy unit of user-day with <span class="math">\(\varepsilon_A=0.2\)</span>.</li>
<li>Process <span class="math">\(B\)</span> uses a privacy unit of user-month with <span class="math">\(\varepsilon_B=3\)</span>.</li>
</ul>
<p>Can we simply multiply <span class="math">\(\varepsilon_A\)</span> by <span class="math">\(30\)</span> to compare it to <span class="math">\(\varepsilon_B\)</span>?
Well, not really. The data of a user <em>during a single day</em> is protected by
Process <span class="math">\(A\)</span> with <span class="math">\(\varepsilon_A\)</span>, which is better than what Process <span class="math">\(B\)</span> can
guarantee (at most <span class="math">\(\varepsilon_B\)</span>). But with process <span class="math">\(A\)</span>, the data <em>of an
entire month</em> is only protected with <span class="math">\(30\varepsilon_A=6\)</span> with Process <span class="math">\(A\)</span>, so
Process <span class="math">\(B\)</span> has better guarantees. And this is without the possibility of using
better privacy accounting methods, to get tighter parameters for the monthly
guarantees of Process <span class="math">\(A\)</span>.</p>
<hr>
<p><small></p>
<p>Thanks to Ashwin Machanavajjhala, Erik Taubenek, Kai Yao, Lars Vilhuber,
Lorraine Wong, Marc Paré, Rodrigo Racanicci, Sergey Yekhanin, and Tancrède
Lepoint for their helpful comments and suggestions.</p>
<p></small></p>
<script type="text/javascript">
var defaultButton = 'More details';
var buttons = document.getElementsByClassName('toggleDetails');
for (var i = 0; i < buttons.length; i++) {
buttons[i].innerHTML = defaultButton;
buttons[i].addEventListener('click', function (event) {
this.innerHTML = this.innerHTML == defaultButton ? 'Fewer details' : defaultButton;
details = this.parentElement.nextElementSibling;
details.style.display = details.style.display == 'none' ? 'block' : 'none';
});
}
</script>
<div class="footnote">
<hr>
<ol>
<li id="fn:googledp">
<p>The project name in the GitHub repository is "Google's differential
privacy libraries"; most of the academic literature uses "GoogleDP" to refer
to it, so I reuse the abbreviation here. <a class="footnote-backref" href="#fnref:googledp" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>A friendly, non-technical introduction to differential privacy2021-09-27T00:00:00+02:002023-09-23T00:00:00+02:00Damien Desfontainestag:desfontain.es,2021-09-27:/privacy/friendly-intro-to-differential-privacy.html<p>An introduction and table of contents for my beginner-friendly blog post series about differential privacy.</p><p><strong>Differential</strong> privacy is getting a lot of attention lately. Companies and
governments are starting to <a href="real-world-differential-privacy.html">publish data</a> anonymized with this
notion. Universities are offering <a href="http://www.gautamkamath.com/CS860-fa2020.html">courses</a> about it. Statisticians are getting
acquainted with this new approach to protecting data. Open-source organizations
are publishing tooling to make differential privacy easier to use.</p>
<p>So, you might be wondering: what's the hype all about? What even is differential
privacy? What makes it so special? How does it work in practice? And, perhaps
more importantly, can I understand it without having to read a bunch of
complicated equations?</p>
<p>The good news is: you've come to the right place. Welcome to my friendly blog
post series about differential privacy! It provides simple explanations for the
core concepts behind differential privacy. It is meant for a wide, non-technical
audience: it doesn't assume any prior knowledge, uses as little math as
possible, and illustrates everything with simple examples and diagrams.</p>
<p><center>
<svg
role="img"
aria-label="Same diagram as before, duplicated, with the bottom line missing one person in the database. A double arrow labeled "basically the same" points to the two outputs."
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
version="1.1"
viewBox="0 0 734.41882 358.09845"
stroke-miterlimit="10"
id="svg4123"
sodipodi:docname="full-diagram.svg"
inkscape:version="0.92.2pre0 (973e216, 2017-07-25)"
width="734.41882"
height="358.09845"
style="fill:none;stroke:none;stroke-linecap:square;stroke-miterlimit:10">
<title>Same diagram as before, duplicated, with the bottom line missing one person in the database. A double arrow labeled "basically the same" points to the two outputs.</title>
<desc>Same diagram as before, duplicated, with the bottom line missing one person in the database. A double arrow labeled "basically the same" points to the two outputs.</desc>
<metadata
id="metadata4129">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
<dc:title></dc:title>
</cc:Work>
</rdf:RDF>
</metadata>
<defs
id="defs4127">
<clipPath
id="p.3-4">
<path
d="M 0,0 H 275 V 252 H 0 Z"
id="path4077-0"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
</defs>
<sodipodi:namedview
pagecolor="#ffffff"
bordercolor="#666666"
borderopacity="1"
objecttolerance="10"
gridtolerance="10"
guidetolerance="10"
inkscape:pageopacity="0"
inkscape:pageshadow="2"
inkscape:window-width="1918"
inkscape:window-height="1078"
id="namedview4125"
showgrid="false"
inkscape:zoom="1.3111111"
inkscape:cx="532.33439"
inkscape:cy="127.30279"
inkscape:window-x="0"
inkscape:window-y="0"
inkscape:window-maximized="0"
inkscape:current-layer="g4121"
fit-margin-top="0"
fit-margin-left="0"
fit-margin-right="0"
fit-margin-bottom="0"
viewbox-x="82"
viewbox-width="650" />
<clipPath
id="p.0">
<path
d="M 0,0 H 960 V 720 H 0 Z"
id="path4036"
inkscape:connector-curvature="0"
style="clip-rule:nonzero" />
</clipPath>
<g
clip-path="url(#p.0)"
id="g4121"
transform="translate(-76.086586,-124.34907)">
<path
d="M 0,0 H 960 V 720 H 0 Z"
id="path4039"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 76.08921,124.34908 h 223.2992 V 267.11813 H 76.08921 Z"
id="path4041"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<g
transform="matrix(0.22307612,0,0,0.22307664,76.089214,124.34908)"
id="g4048">
<clipPath
id="p.1">
<path
d="M 0,0 H 1001 V 640 H 0 Z"
id="path4043"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
<image
clip-path="url(#p.1)"
width="1001"
height="640"
x="0"
y="0"
preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/quiestce.jpeg"
id="image4046"
style="fill:#000000" />
</g>
<path
d="M 76.086586,339.67847 H 299.38579 V 482.44751 H 76.086586 Z"
id="path4050"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<g
transform="matrix(0.22307612,0,0,0.22307664,76.086589,339.67847)"
id="g4057">
<clipPath
id="p.2">
<path
d="M 0,0 H 1001 V 640 H 0 Z"
id="path4052"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
<image
clip-path="url(#p.2)"
width="1001"
height="640"
x="0"
y="0"
preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/quiestce.jpeg"
id="image4055"
style="fill:#000000" />
</g>
<path
d="m 604.81974,178.70273 v 0 c -1.51233,-10.41394 3.45307,-20.72311 12.78925,-26.55296 9.33618,-5.82984 21.40576,-6.15791 31.08722,-0.84502 v 0 c 3.42944,-6.0551 9.70636,-10.23573 16.93207,-11.27733 7.2257,-1.04159 14.55151,1.17818 19.76159,5.98789 v 0 c 2.92145,-5.48995 8.65778,-9.17848 15.1734,-9.75675 6.51569,-0.57824 12.88855,2.03563 16.85718,6.9141 v 0 c 5.27802,-5.81934 13.67554,-8.2695 21.55884,-6.29033 7.8833,1.97917 13.83649,8.03222 15.28363,15.53996 v 0 c 6.46649,1.65271 11.8529,5.85413 14.76764,11.51872 2.91473,5.66462 3.07178,12.23673 0.43066,18.01831 v 0 c 6.36756,7.76532 7.85712,18.11251 3.91272,27.18018 -3.9444,9.06767 -12.7301,15.49351 -23.07849,16.87953 -0.0729,8.51033 -5.05414,16.31931 -13.02356,20.41701 -7.96942,4.09772 -17.68268,3.84427 -25.39587,-0.66263 -3.2854,10.1926 -12.53272,17.6922 -23.74671,19.25873 -11.21405,1.56653 -22.38446,-3.08087 -28.68524,-11.93434 -7.72339,4.3639 -16.99084,5.621 -25.71185,3.48773 -8.72095,-2.13329 -16.16077,-7.47725 -20.64124,-14.8264 v 0 c -7.89233,0.86539 -15.52319,-2.96604 -19.10534,-9.59276 -3.58216,-6.62671 -2.35303,-14.63803 3.07733,-20.05798 v 0 c -7.04022,-3.8826 -10.63251,-11.58696 -8.90375,-19.09555 1.72882,-7.50861 8.38696,-13.11998 16.50256,-13.90799 z"
id="path4059"
inkscape:connector-curvature="0"
style="fill:#f6cd4c;fill-rule:evenodd" />
<path
d="m 597.06497,212.10764 v 0 c 3.32227,1.83222 7.16028,2.66335 10.99872,2.38181 m 5.02759,27.2696 c 1.65063,-0.18098 3.26861,-0.56427 4.81219,-1.13993 m 41.53858,12.47812 c -1.16095,-1.63134 -2.133,-3.37456 -2.89954,-5.19996 m 55.3335,-2.12474 v 0 c 0.59893,-1.85826 0.98706,-3.77081 1.15777,-5.70575 m 37.26019,-14.04791 c 0.0776,-9.06056 -5.41461,-17.3565 -14.11767,-21.32448 m 33.2832,-22.73425 c -1.40942,3.08528 -3.56109,5.8222 -6.28625,7.99617 m -8.91071,-37.53403 v 0 c 0.24011,1.24591 0.35126,2.51055 0.33185,3.77639 m -37.17371,-13.02565 v 0 c -1.31659,1.45164 -2.4013,3.07382 -3.22033,4.81604 m -28.81055,-1.97402 v 0 c -0.70166,1.31851 -1.22559,2.71372 -1.55975,4.15347 m -35.13446,1.13608 v 0 c 2.04858,1.12421 3.94384,2.47731 5.64404,4.02959 m -49.5199,23.36855 v 0 c 0.20844,1.4353 0.53784,2.85292 0.98499,4.23957"
id="path4061"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 636.30257,211.8428 h -0.1875 q -0.78125,0 -1.25,-0.48437 -0.46875,-0.48438 -0.46875,-1.17188 0,-0.46875 0.5,-2.92187 l 1.375,-7.10938 q 0.46875,-2.42187 2.0625,-13.4375 l 0.39063,-2.70312 q 0.14062,-1.01563 1.10937,-1.84375 0.96875,-0.84375 1.78125,-0.84375 0.4375,0 2.9375,1.3125 2.70313,1.45312 3.20313,1.59375 5.09375,1.875 8.375,5.6875 3.29687,3.8125 3.29687,8.23437 0,4.04688 -2.40625,7.92188 -2.39062,3.875 -5.78125,5.71875 -3.39062,1.84375 -8.67187,1.84375 -1.53125,0 -3.46875,-0.53125 -1.9375,-0.51563 -2.79688,-1.26563 z m 6,-25.40625 -1.64062,11.20313 -2,10.67187 q 0.0625,0.0469 0.14062,0.0781 1.84375,1.32812 4.375,1.32812 5.01563,0 7.375,-1.79687 2.35938,-1.79688 3.60938,-4.21875 1.25,-2.42188 1.25,-4.92188 0,-2.51562 -1.40625,-4.64062 -1.39063,-2.125 -3.76563,-3.57813 -2.35937,-1.46875 -7.9375,-4.125 z m 32.16406,24.28125 q -4.21875,2.03125 -6.01562,2.03125 -7.29688,0 -7.29688,-6.95312 0,-6.46875 4.07813,-10.46875 4.09375,-4 8.98437,-4 1.89063,0 3.90625,0.96875 2.03125,0.96875 2.03125,2.25 0,0.79687 -0.53125,1.25 -0.39062,0.76562 -1.10937,4.40625 -0.70313,3.64062 -0.70313,5.25 0,1.96875 1.01563,5.42187 l 0.0781,0.3125 q -0.4375,1.82813 -2.17187,1.82813 -0.29688,0 -1.09375,-0.79688 -0.79688,-0.79687 -1.17188,-1.5 z m 1.42188,-15.1875 q -1.21875,-0.73437 -1.95313,-0.73437 -3.25,0 -6.07812,2.95312 -2.82813,2.9375 -2.82813,7.3125 0,4.29688 3.71875,4.29688 2.90625,0 5.46875,-2.15625 0.3125,-8.1875 1.67188,-11.67188 z m 16.45312,-0.34375 q -0.23437,0.9375 -0.60937,2.1875 -1.64063,5.42188 -1.64063,8.26563 0,3.35937 1.5625,3.35937 1.54688,0 2.28125,-0.95312 1.01563,-1.32813 1.875,-1.32813 0.65625,0 1.25,0.48438 0.59375,0.48437 0.59375,1.26562 0,1.71875 -2.15625,2.95313 -2.15625,1.21875 -4.29687,1.21875 -5.03125,0 -5.03125,-6.28125 0,-3.4375 1.54687,-8.39063 0.40625,-1.3125 0.78125,-2.625 -4.25,-0.20312 -4.85937,-0.70312 -0.60938,-0.5 -0.60938,-1.5625 0,-0.85938 0.57813,-1.42188 0.59375,-0.57812 1.60937,-0.57812 l 4.03125,0.32812 q 0.125,-0.51562 0.1875,-1.17187 0.0312,-0.67188 0.125,-1.48438 l 0.34375,-2.42187 q 0.10938,-0.79688 0.71875,-1.34375 0.625,-0.54688 1.46875,-0.54688 1.64063,0 1.64063,2.625 0,0.76563 -0.23438,1.98438 l -0.39062,2.35937 q 2.60937,-0.32812 2.78125,-0.32812 2.03125,0 2.79687,0.40625 0.76563,0.39062 0.76563,1.46875 0,0.95312 -0.65625,1.54687 -0.65625,0.57813 -1.57813,0.57813 l -1.92187,-0.0781 q -1.45313,0 -2.95313,0.1875 z m 21.44141,15.53125 q -4.21875,2.03125 -6.01563,2.03125 -7.29687,0 -7.29687,-6.95312 0,-6.46875 4.07812,-10.46875 4.09375,-4 8.98438,-4 1.89062,0 3.90625,0.96875 2.03125,0.96875 2.03125,2.25 0,0.79687 -0.53125,1.25 -0.39063,0.76562 -1.10938,4.40625 -0.70312,3.64062 -0.70312,5.25 0,1.96875 1.01562,5.42187 l 0.0781,0.3125 q -0.4375,1.82813 -2.17188,1.82813 -0.29687,0 -1.09375,-0.79688 -0.79687,-0.79687 -1.17187,-1.5 z m 1.42187,-15.1875 q -1.21875,-0.73437 -1.95312,-0.73437 -3.25,0 -6.07813,2.95312 -2.82812,2.9375 -2.82812,7.3125 0,4.29688 3.71875,4.29688 2.90625,0 5.46875,-2.15625 0.3125,-8.1875 1.67187,-11.67188 z"
id="path4063"
inkscape:connector-curvature="0"
style="fill:#000000;fill-rule:nonzero" />
<path
d="m 604.81711,394.03214 v 0 c -1.51233,-10.41397 3.45307,-20.72311 12.78925,-26.55298 9.33618,-5.82983 21.40576,-6.1579 31.08722,-0.845 v 0 c 3.42944,-6.05511 9.70636,-10.23575 16.93207,-11.27734 7.2257,-1.0416 14.55151,1.17819 19.76159,5.98788 v 0 c 2.92145,-5.48993 8.65778,-9.17847 15.1734,-9.75674 6.51569,-0.57825 12.88855,2.03564 16.85718,6.91409 v 0 c 5.27802,-5.81934 13.67554,-8.2695 21.55884,-6.29031 7.8833,1.97915 13.83649,8.03222 15.28363,15.53994 v 0 c 6.46649,1.65271 11.8529,5.85413 14.76764,11.51874 2.91473,5.66461 3.07178,12.23673 0.43066,18.01831 v 0 c 6.36756,7.76532 7.85712,18.11249 3.91272,27.18018 -3.9444,9.06766 -12.7301,15.4935 -23.07849,16.87952 -0.0729,8.51034 -5.05414,16.3193 -13.02356,20.41702 -7.96942,4.09769 -17.68268,3.84427 -25.39587,-0.66266 -3.2854,10.19263 -12.53272,17.69223 -23.74671,19.25876 -11.21405,1.56653 -22.38446,-3.08087 -28.68524,-11.93436 -7.72339,4.36392 -16.99084,5.621 -25.71185,3.48773 -8.72095,-2.13327 -16.16077,-7.47723 -20.64124,-14.82638 v 0 c -7.89233,0.86539 -15.52319,-2.96607 -19.10534,-9.59277 -3.58216,-6.62671 -2.35303,-14.63804 3.07733,-20.05799 v 0 c -7.04022,-3.8826 -10.63257,-11.58694 -8.90375,-19.09555 1.72882,-7.5086 8.38696,-13.11996 16.50256,-13.90799 z"
id="path4065"
inkscape:connector-curvature="0"
style="fill:#f6cd4c;fill-rule:evenodd" />
<path
d="m 597.06237,427.43705 v 0 c 3.32227,1.83218 7.16028,2.66333 10.99872,2.38181 m 5.02759,27.26959 c 1.65063,-0.181 3.26861,-0.56427 4.81219,-1.13992 m 41.53858,12.47811 c -1.16095,-1.63134 -2.133,-3.37457 -2.89954,-5.19998 m 55.3335,-2.12472 v 0 c 0.59893,-1.85825 0.98706,-3.77082 1.15777,-5.70575 m 37.26019,-14.04792 c 0.0776,-9.06054 -5.41461,-17.3565 -14.11767,-21.32449 m 33.2832,-22.73425 c -1.40942,3.0853 -3.56109,5.82223 -6.28625,7.99618 m -8.91071,-37.53402 v 0 c 0.24011,1.24591 0.35126,2.51056 0.33185,3.77639 m -37.17371,-13.02566 v 0 c -1.31659,1.45163 -2.4013,3.07382 -3.22033,4.81604 m -28.81055,-1.974 v 0 c -0.70166,1.31848 -1.22559,2.71372 -1.55975,4.15347 m -35.13446,1.13608 v 0 c 2.04858,1.12421 3.94384,2.4773 5.64404,4.02957 m -49.5199,23.36856 v 0 c 0.20844,1.43531 0.53784,2.85291 0.98499,4.23957"
id="path4067"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 636.29997,427.17219 h -0.1875 q -0.78125,0 -1.25,-0.48437 -0.46875,-0.48438 -0.46875,-1.17188 0,-0.46875 0.5,-2.92187 l 1.375,-7.10938 q 0.46875,-2.42187 2.0625,-13.4375 l 0.39063,-2.70312 q 0.14062,-1.01563 1.10937,-1.84375 0.96875,-0.84375 1.78125,-0.84375 0.4375,0 2.9375,1.3125 2.70313,1.45312 3.20313,1.59375 5.09375,1.875 8.375,5.6875 3.29687,3.8125 3.29687,8.23437 0,4.04688 -2.40625,7.92188 -2.39062,3.875 -5.78125,5.71875 -3.39062,1.84375 -8.67187,1.84375 -1.53125,0 -3.46875,-0.53125 -1.9375,-0.51563 -2.79688,-1.26563 z m 6,-25.40625 -1.64062,11.20313 -2,10.67187 q 0.0625,0.0469 0.14062,0.0781 1.84375,1.32812 4.375,1.32812 5.01563,0 7.375,-1.79687 2.35938,-1.79688 3.60938,-4.21875 1.25,-2.42188 1.25,-4.92188 0,-2.51562 -1.40625,-4.64062 -1.39063,-2.125 -3.76563,-3.57813 -2.35937,-1.46875 -7.9375,-4.125 z m 32.16406,24.28125 q -4.21875,2.03125 -6.01562,2.03125 -7.29688,0 -7.29688,-6.95312 0,-6.46875 4.07813,-10.46875 4.09375,-4 8.98437,-4 1.89063,0 3.90625,0.96875 2.03125,0.96875 2.03125,2.25 0,0.79687 -0.53125,1.25 -0.39062,0.76562 -1.10937,4.40625 -0.70313,3.64062 -0.70313,5.25 0,1.96875 1.01563,5.42187 l 0.0781,0.3125 q -0.4375,1.82813 -2.17187,1.82813 -0.29688,0 -1.09375,-0.79688 -0.79688,-0.79687 -1.17188,-1.5 z m 1.42188,-15.1875 q -1.21875,-0.73437 -1.95313,-0.73437 -3.25,0 -6.07812,2.95312 -2.82813,2.9375 -2.82813,7.3125 0,4.29688 3.71875,4.29688 2.90625,0 5.46875,-2.15625 0.3125,-8.1875 1.67188,-11.67188 z m 16.45312,-0.34375 q -0.23437,0.9375 -0.60937,2.1875 -1.64063,5.42188 -1.64063,8.26563 0,3.35937 1.5625,3.35937 1.54688,0 2.28125,-0.95312 1.01563,-1.32813 1.875,-1.32813 0.65625,0 1.25,0.48438 0.59375,0.48437 0.59375,1.26562 0,1.71875 -2.15625,2.95313 -2.15625,1.21875 -4.29687,1.21875 -5.03125,0 -5.03125,-6.28125 0,-3.4375 1.54687,-8.39063 0.40625,-1.3125 0.78125,-2.625 -4.25,-0.20312 -4.85937,-0.70312 -0.60938,-0.5 -0.60938,-1.5625 0,-0.85938 0.57813,-1.42188 0.59375,-0.57812 1.60937,-0.57812 l 4.03125,0.32812 q 0.125,-0.51562 0.1875,-1.17187 0.0312,-0.67188 0.125,-1.48438 l 0.34375,-2.42187 q 0.10938,-0.79688 0.71875,-1.34375 0.625,-0.54688 1.46875,-0.54688 1.64063,0 1.64063,2.625 0,0.76563 -0.23438,1.98438 l -0.39062,2.35937 q 2.60937,-0.32812 2.78125,-0.32812 2.03125,0 2.79687,0.40625 0.76563,0.39062 0.76563,1.46875 0,0.95312 -0.65625,1.54687 -0.65625,0.57813 -1.57813,0.57813 l -1.92187,-0.0781 q -1.45313,0 -2.95313,0.1875 z m 21.44141,15.53125 q -4.21875,2.03125 -6.01563,2.03125 -7.29687,0 -7.29687,-6.95312 0,-6.46875 4.07812,-10.46875 4.09375,-4 8.98438,-4 1.89062,0 3.90625,0.96875 2.03125,0.96875 2.03125,2.25 0,0.79687 -0.53125,1.25 -0.39063,0.76562 -1.10938,4.40625 -0.70312,3.64062 -0.70312,5.25 0,1.96875 1.01562,5.42187 l 0.0781,0.3125 q -0.4375,1.82813 -2.17188,1.82813 -0.29687,0 -1.09375,-0.79688 -0.79687,-0.79687 -1.17187,-1.5 z m 1.42187,-15.1875 q -1.21875,-0.73437 -1.95312,-0.73437 -3.25,0 -6.07813,2.95312 -2.82812,2.9375 -2.82812,7.3125 0,4.29688 3.71875,4.29688 2.90625,0 5.46875,-2.15625 0.3125,-8.1875 1.67187,-11.67188 z"
id="path4069"
inkscape:connector-curvature="0"
style="fill:#000000;fill-rule:nonzero" />
<path
d="m 299.38846,189.04987 h 70.36065 v -6.68504 l 8.40525,13.37008 -8.40525,13.37007 v -6.68502 h -70.36065 z"
id="path4071"
inkscape:connector-curvature="0"
style="fill:#7c7ce0;fill-rule:evenodd;stroke-width:0.79288208" />
<path
d="M 427.88773,353.66272 H 553.16596 V 468.46329 H 427.88773 Z"
id="path4075"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<g
transform="matrix(0.45555722,0,0,0.45555774,379.83689,353.66273)"
id="g4082">
<clipPath
id="p.3">
<path
d="M 0,0 H 275 V 252 H 0 Z"
id="path4077"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
<image
clip-path="url(#p.3)"
width="275"
height="252"
x="0"
y="0"
preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/magic.gif"
id="image4080"
style="fill:#000000" />
</g>
<path
d="m 660.00147,264.20093 21.66931,-21.66928 21.66932,21.66928 h -10.83466 v 85.7638 h 10.83466 l -21.66932,21.66928 -21.66931,-21.66928 h 10.83466 v -85.7638 z"
id="path4088"
inkscape:connector-curvature="0"
style="fill:#db4437;fill-rule:evenodd" />
<path
d="m 660.00147,264.20093 21.66931,-21.66928 21.66932,21.66928 h -10.83466 v 85.7638 h 10.83466 l -21.66932,21.66928 -21.66931,-21.66928 h 10.83466 v -85.7638 z"
id="path4090"
inkscape:connector-curvature="0"
style="fill-rule:evenodd;stroke:#666666;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
<path
d="m 554.53672,285.04017 c 42.66144,-12.09317 85.32288,12.0932 127.98438,0 42.66143,-12.09317 85.32287,12.0932 127.98431,0 l -1.70062,50.79135 c -42.66144,12.09317 -85.32288,-12.09317 -127.98432,0 -42.66143,12.09317 -85.32294,-12.09317 -127.98437,0 z"
id="path4092"
inkscape:connector-curvature="0"
style="fill:#db4437;fill-rule:evenodd" />
<path
d="m 587.72404,321.45273 q -0.84375,-0.5 -1.26562,-1.0625 -0.40625,-0.5625 -0.40625,-1.07812 0,-0.67188 0.67187,-1.07813 0.125,-0.0625 0.35938,-0.0625 0.3125,0 0.67187,0.10938 0.375,0.10937 0.57813,0.29687 0.82812,0.76563 1.98437,0.76563 0.5,0 1.46875,-0.21875 1.25,-0.39063 1.96875,-0.95313 0.71875,-0.57812 1.70313,-1.79687 0.60937,-0.79688 0.90625,-1.60938 0.3125,-0.82812 0.3125,-1.5 0,-0.73437 -0.34375,-1.15625 -0.64063,-0.85937 -1.4375,-1.28125 -0.78125,-0.4375 -1.82813,-0.4375 -0.9375,0 -1.82812,0.25 l -1.85938,0.39063 q -0.125,0.0312 -0.34375,0.0469 -0.20312,0.0156 -0.59375,0.95313 -0.375,0.92187 -1.29687,3.51562 -0.67188,1.875 -1.125,2.70313 -0.45313,0.8125 -0.79688,0.8125 -0.39062,0 -0.82812,-0.70313 -0.0937,-0.15625 -0.40625,-0.64062 -0.29688,-0.48438 -0.29688,-0.9375 0,-0.21875 0.15625,-0.53125 0.15625,-0.29688 0.26563,-0.57813 0.10937,-0.29687 0.25,-0.79687 0.40625,-1.25 2.40625,-6.20313 2,-4.96875 3.125,-7.5 0.79687,-1.82812 0.48437,-1.92187 0,-0.0937 -0.10937,-0.28125 -0.0937,-0.20313 -0.0625,-0.29688 0,-0.1875 1.14062,-0.5625 1.14063,-0.39062 1.54688,-0.32812 h 0.20312 q 0.40625,0 0.40625,-0.15625 0.0625,0 0.59375,-0.0625 0.53125,-0.0625 1.14063,0 2.26562,0.1875 3.57812,1.29687 1.3125,1.09375 1.28125,3.23438 -0.0625,0.84375 -0.92187,2.28125 -0.85938,1.4375 -1.89063,2.35937 -0.51562,0.45313 -1.17187,0.95313 -0.65625,0.5 -0.84375,0.625 l 0.79687,0.28125 q 1.5,0.57812 2.32813,1.42187 0.84375,0.82813 1.1875,2.26563 0.0781,0.3125 0.0781,0.57812 0,0.70313 -0.25,1.59375 -0.23437,0.89063 -0.53125,1.5625 -0.1875,0.20313 -0.3125,0.42188 -0.125,0.21875 -0.15625,0.28125 -0.0312,0.14062 -0.57812,0.98437 -0.54688,0.84375 -0.96875,1.35938 -2.10938,1.79687 -3.48438,2.29687 -1.34375,0.60938 -3.20312,0.60938 -1.65625,0 -2.45313,-0.51563 z m 2.29688,-13.26562 q 0.32812,-0.0781 1.28125,-0.28125 0.95312,-0.20313 1.375,-0.35938 0.0625,-0.0312 0.39062,-0.20312 0.34375,-0.1875 0.70313,-0.54688 2.01562,-1.875 2.8125,-3.39062 0.79687,-1.3125 0.79687,-1.84375 0,-0.39063 -0.34375,-0.75 -0.60937,-0.70313 -2.01562,-0.70313 -0.60938,0 -1.03125,0.14063 -0.45313,0.125 -0.65625,0.29687 -0.20313,0.17188 -0.20313,0.53125 0,0.25 -0.17187,0.5625 -0.15625,0.29688 -0.34375,0.48438 -0.0312,0.20312 -0.17188,0.39062 -0.14062,0.1875 -0.21875,0.25 0,0.0781 -0.4375,0.875 -0.42187,0.79688 -0.79687,1.78125 -0.96875,2.28125 -0.96875,2.76563 z m 11.90643,10.29687 q -0.0937,-0.1875 -0.54688,-0.5 -0.45312,-0.32812 -0.45312,-0.39062 0,-0.14063 -0.14063,-0.375 -0.14062,-0.23438 -0.26562,-0.4375 -0.26563,-0.25 0.34375,-1.65625 0.60937,-1.40625 1.64062,-3.32813 0.1875,-0.1875 0.48438,-0.67187 0.3125,-0.48438 0.40625,-0.54688 0,-0.125 0.0937,-0.23437 0.0937,-0.10938 0.23437,-0.10938 l 0.25,-0.51562 q 0.125,-0.29688 1.25,-1.375 1.125,-1.09375 1.76563,-1.46875 0.59375,-0.42188 1.3125,-0.42188 0.76562,0 1.29687,0.45313 l 0.875,0.73437 q 0.95313,0.79688 1.07813,0.92188 l 1.1875,1.0625 -0.54688,1.59375 q -0.28125,0.85937 -0.28125,1.64062 0,1.20313 0.67188,2.26563 0.15625,0.34375 0.42187,0.54687 0.28125,0.1875 0.73438,0.25 0.70312,0.125 0.70312,0.70313 0,0.70312 -0.26562,1.0625 -0.25,0.34375 -0.60938,0.4375 -0.5,0.0781 -0.70312,0.0781 -0.92188,0 -1.6875,-0.48437 -0.76563,-0.48438 -1.34375,-1.64063 -0.125,-0.1875 -0.34375,-0.6875 -0.20313,-0.51562 -0.23438,-0.71875 -0.21875,0.32813 -0.73437,0.84375 -3.29688,3.32813 -5.34375,3.32813 -0.70313,0 -1.25,-0.35938 z m 1.76562,-2.65625 q 0.5,0.0625 1.98438,-1.15625 0.15625,-0.125 0.34375,-0.29687 0.1875,-0.17188 0.42187,-0.375 0.89063,-0.73438 1.39063,-1.21875 0.5,-0.5 1,-1.20313 l 0.70312,-1.15625 q -0.3125,-0.70312 -0.57812,-0.9375 -0.25,-0.25 -0.5,-0.25 -0.64063,0 -1.60938,1.14063 -0.95312,1.14062 -2.29687,3.4375 -0.35938,0.57812 -0.57813,1.20312 -0.21875,0.625 -0.28125,0.8125 z m 14.94391,3.8125 q -0.78125,0 -1.59375,-0.32812 -0.8125,-0.3125 -1.375,-0.78125 -0.5625,-0.46875 -0.5625,-0.84375 0,-0.39063 0.35938,-0.48438 l 0.15625,-0.125 q 0.0625,-0.0312 0.25,-0.0312 0.14062,-0.0312 0.32812,0.0156 0.1875,0.0469 0.3125,0.0469 0.23438,0.0625 0.67188,0.125 l 0.84375,0.0937 q 0.64062,0 1.1875,-0.125 0.5625,-0.125 0.5625,-0.28125 0,-0.10937 -0.25,-0.32812 -0.25,-0.21875 -0.54688,-0.21875 0,-0.125 -0.98437,-0.85938 -0.54688,-0.42187 -1.04688,-0.82812 -0.5,-0.42188 -0.5,-0.625 l 0.0312,-0.0625 v -0.15625 q -0.21875,0 -0.42187,-0.60938 -0.1875,-0.60937 -0.1875,-1.1875 0,-1.07812 0.95312,-2.39062 0.96875,-1.3125 2.39063,-2.35938 1.4375,-1.0625 2.71875,-1.45312 0.46875,-0.21875 1.40625,-0.21875 0.60937,0 0.82812,0.125 l 0.25,-0.0625 q 0.39063,0 1,0.64062 0.60938,0.64063 0.76563,1.01563 0,1.0625 -0.0625,1.70312 -0.0625,0.64063 -0.32813,0.64063 -0.0312,0 -0.0937,0.0312 -0.0625,0.0312 -0.0625,0.15625 0,0.45312 -0.78125,1 -0.78125,0.53125 -1.42187,0.53125 l -0.125,-0.0312 q -0.20313,-0.0312 -0.57813,-0.375 -0.35937,-0.35938 -0.35937,-0.70313 0,-0.0937 0.375,-0.51562 0.39062,-0.42188 0.65625,-0.8125 0.28125,-0.40625 0.28125,-0.85938 0,-0.15625 -0.10938,-0.20312 -0.10937,-0.0469 -0.4375,-0.0469 -1.75,0.79688 -2.51562,1.5 -0.10938,0.20313 -0.1875,0.29688 -0.0781,0.0937 -0.20313,0.0937 0,0 0,-0.0156 0,-0.0156 -0.0312,-0.0156 -0.0625,0 -0.42187,0.375 -0.34375,0.35937 -0.65625,0.84375 -0.29688,0.46875 -0.35938,0.89062 l -0.0625,0.0937 v 0.15625 q -0.0312,0.0625 -0.0312,0.26563 0,0.28125 0.25,0.59375 0.25,0.29687 1.03125,0.84375 1.4375,1.09375 1.95313,1.625 0.53125,0.51562 0.67187,0.9375 0.28125,0.57812 0.28125,1.01562 0,1.09375 -1.14062,1.6875 -1.125,0.59375 -3.07813,0.59375 z m 13.21564,-13.79687 q -0.10937,0.0937 -0.20312,0.0937 -0.125,0 -0.40625,-0.1875 -0.32813,-0.0312 -0.85938,-0.54688 -0.51562,-0.51562 -0.51562,-0.67187 -0.0312,-0.39063 0.0625,-1.10938 0.0937,-0.71875 0.35937,-1.28125 0.28125,-0.57812 0.71875,-0.60937 0.23438,0.0312 0.60938,0.14062 0.39062,0.10938 0.875,0.29688 0.59375,1.21875 0.59375,2.07812 0,1.28125 -1.23438,1.79688 z m -2.95312,13.28125 q -0.76563,0 -1.32813,-0.79688 -0.5625,-0.79687 -0.6875,-1.98437 0.125,-1.1875 0.1875,-1.73438 0.35938,-1.5625 0.89063,-3.6875 0.53125,-2.14062 0.625,-2.35937 0.15625,-0.28125 0.5,-0.28125 0.35937,0 0.78125,0.26562 0.4375,0.26563 0.73437,0.75 0.25,0.35938 0.25,0.79688 0,0.67187 -0.64062,2.82812 -0.4375,1.59375 -0.57813,2.26563 -0.125,0.67187 -0.0937,1.4375 -0.125,1.4375 -0.25,1.98437 -0.125,0.54688 -0.39062,0.51563 z m 6.8189,-0.21875 q -0.95312,0 -1.25,-0.10938 -0.28125,-0.125 -0.70312,-0.5 -0.0937,-0.23437 -0.40625,-0.60937 -0.29688,-0.39063 -0.48438,-0.57813 -0.0937,-0.35937 -0.0937,-0.82812 0,-0.84375 0.23438,-1.95313 0.23437,-1.125 0.59375,-2.04687 0.125,-0.26563 0.20312,-0.48438 0.0781,-0.21875 0.10938,-0.28125 0,-0.23437 0.8125,-1.39062 0.82812,-1.17188 1.3125,-1.625 0.21875,-0.25 0.67187,-0.53125 0.46875,-0.29688 0.82813,-0.42188 0.0937,-0.0312 0.78125,-0.28125 0.6875,-0.26562 1.26562,-0.26562 0.51563,0 1,0.20312 0.5,0.20313 0.82813,0.65625 0.82812,0.82813 0.82812,2.17188 0,0.26562 -0.0625,0.57812 -0.17187,0.64063 -0.67187,1.04688 -0.48438,0.39062 -1.03125,0.39062 -0.54688,0 -0.92188,-0.40625 -0.125,-0.32812 -0.125,-0.42187 0,-0.0937 0.125,-0.54688 0.125,-0.57812 0.125,-0.73437 0,-0.40625 -0.28125,-0.40625 -0.39062,0 -0.625,0.1875 -0.21875,0.1875 -0.82812,0.82812 -0.40625,0.45313 -0.96875,1.29688 -0.5625,0.84375 -0.875,1.60937 -0.29688,0.875 -0.40625,1.25 -0.10938,0.35938 -0.10938,0.42188 0,0.125 -0.0312,0.23437 -0.0312,0.10938 -0.0312,0.25 0,0.21875 0.0937,0.34375 0.0937,0.125 0.34375,0.26563 0.0781,0.0312 0.23438,0.125 0.15625,0.0937 0.3125,0.0937 0.20312,0 0.54687,-0.15625 0.35938,-0.17188 0.45313,-0.20313 0.40625,-0.1875 1.17187,-0.82812 l 0.78125,-0.57813 q 0.21875,-0.15625 0.39063,-0.26562 0.1875,-0.10938 0.28125,-0.15625 0.0937,-0.0469 0.0937,0.0156 0,0.125 0.0937,0.125 0.0937,0 0.21875,-0.125 0.125,-0.20312 0.45312,-0.20312 0.25,0 0.51563,0.15625 0.28125,0.14062 0.28125,0.23437 l -0.0312,0.15625 q 0,0.15625 0.15625,0.32813 0,0.125 -0.0781,0.32812 -0.0781,0.20313 -0.20312,0.34375 -0.32813,0.46875 -0.73438,0.98438 -0.39062,0.51562 -0.54687,0.70312 h -0.0312 q -0.15625,0 -0.45313,0.29688 -0.46875,0.4375 -0.82812,0.4375 -0.125,0.0625 -0.29688,0.15625 -0.15625,0.0781 -0.15625,0.17187 -0.53125,0.54688 -2.875,0.54688 z m 10.59815,-0.32813 q -0.54688,0.125 -1.3125,-0.57812 -0.76563,-0.70313 -0.90625,-1.3125 -0.0625,-0.3125 -0.0625,-0.54688 0,-1.23437 0.79687,-3.01562 0.8125,-1.78125 2.17188,-3.34375 1.35937,-1.5625 2.98437,-2.26563 0.67188,-0.35937 0.85938,-0.35937 0.20312,0 0.875,0.35937 0.64062,0.15625 1.01562,0.34375 0.39063,0.1875 0.70313,0.60938 0.42187,0.82812 0.60937,2.04687 0.20313,0.67188 0.20313,0.95313 0,0.26562 -0.20313,0.45312 -0.0312,0.0312 -0.10937,0.125 -0.0781,0.0937 -0.0781,0.15625 0,0.0469 0.0625,0.0469 0,0.0312 -0.0312,0.14062 -0.0312,0.10938 -0.125,0.29688 -0.39062,1.21875 -0.39062,1.89062 0,0.21875 0.0625,0.54688 0.32812,0.82812 0.8125,0.82812 0.4375,0 1.01562,-0.57812 0.48438,-0.3125 0.96875,-0.3125 0.28125,0 0.28125,0.15625 0.0312,0.0312 0.125,0.0937 0.0937,0.0625 0.1875,0.0625 0.14063,0 0.14063,0.29687 0,0.25 -0.125,0.67188 -0.10938,0.40625 -0.26563,0.67187 -0.21875,0.21875 -0.35937,0.39063 -0.125,0.15625 -0.125,0.25 l -0.46875,0.4375 q -0.60938,0.35937 -1.45313,0.35937 -0.85937,0 -1.375,-0.35937 -0.5,-0.34375 -0.95312,-0.875 -0.45313,-0.53125 -0.57813,-0.9375 l -0.15625,-0.32813 -0.9375,0.60938 -0.79687,0.82812 q -0.25,0.32813 -0.4375,0.32813 h -0.32813 q 0,0.21875 -0.90625,0.54687 -0.89062,0.3125 -1.39062,0.3125 z m 0.67187,-2.6875 q 0.34375,0 1.4375,-0.8125 1.10938,-0.8125 1.5,-1.29687 0.125,-0.15625 0.32813,-0.45313 0.21875,-0.3125 0.375,-0.46875 0.51562,-0.60937 0.75,-0.96875 0.25,-0.375 0.25,-0.73437 0,-0.25 -0.23438,-0.70313 -0.125,-0.39062 -0.375,-0.39062 -0.14062,0 -0.73437,0.23437 -0.60938,0.34375 -1.4375,1.4375 -0.8125,1.07813 -1.42188,2.20313 -0.60937,1.125 -0.60937,1.64062 0,0.0937 0.0625,0.1875 0.0781,0.0937 0.10937,0.125 z m 11.06427,4.21875 q -0.48437,-0.15625 -0.71875,-0.57812 -0.23437,-0.40625 -0.23437,-1.20313 0,-1.3125 0.25,-2.5625 l 0.0312,-0.25 q 0.0312,-0.20312 0.0937,-0.625 0.0625,-0.4375 0.0937,-0.25 l 0.20312,-0.85937 q 0.21875,-0.89063 0.42188,-1.65625 0.21875,-0.78125 0.34375,-1.125 l 0.34375,-1.125 q 0.96875,-3.10938 1.79687,-4.1875 0.0312,-0.125 0.0937,-0.23438 0.0625,-0.125 0.0937,-0.25 0,-0.51562 0.20313,-0.79687 0.28125,-0.48438 0.46875,-0.85938 l 0.14062,-0.35937 0.89063,-1.82813 q 0.0937,-0.3125 0.39062,-0.6875 0.3125,-0.39062 0.625,-0.53125 l -0.0312,-0.0312 q 0.17188,-0.25 0.59375,-0.4375 0.4375,-0.20312 0.75,-0.20312 0.42188,0 0.67188,0.26562 0.26562,0.25 0.26562,0.70313 v 0.3125 q 0,0.29687 -0.0469,0.45312 -0.0469,0.15625 -0.21875,0.39063 h 0.0312 q 0.0469,0.0312 -0.0469,0.20312 -0.0781,0.17188 -0.14062,0.32813 -0.21875,0.57812 -0.3125,0.64062 -0.23438,0.29688 -1.48438,2.71875 -1.23437,2.40625 -1.59375,3.32813 l -0.125,0.48437 -0.0625,0.20313 q -0.35937,0.76562 -0.39062,0.82812 l -0.51563,1.98438 -0.125,0.67187 -0.28125,0.73438 -0.1875,0.60937 q -0.0781,0.15625 -0.125,0.42188 -0.0469,0.25 -0.10937,0.3125 l -0.0312,0.15625 q -0.0312,0.125 -0.21875,0.73437 -0.29688,0.90625 -0.42188,1.5 -0.125,0.57813 -0.15625,1.25 l -0.0312,0.14063 q -0.0937,0.70312 -0.40625,1.04687 -0.29687,0.35938 -0.78125,0.21875 z m 6.82099,-0.59375 q -0.0937,0.0312 -0.26563,0.0312 -0.67187,0 -1.125,-0.76562 -0.4375,-0.78125 -0.4375,-1.96875 0,-0.625 0.125,-1.23438 0.35938,-0.9375 0.35938,-1.34375 0.0937,-0.60937 0.65625,-2.1875 0.5625,-1.59375 0.71875,-1.92187 0.125,0 0.21875,-0.20313 0.0937,-0.20312 0.0937,-0.46875 0,-0.28125 0.10937,-0.57812 0.125,-0.3125 0.25,-0.4375 0,-0.0937 0.0781,-0.34375 0.0781,-0.26563 0.0781,-0.35938 0,-0.0625 0.0781,-0.23437 0.0781,-0.1875 0.0781,-0.28125 0.51563,-0.98438 1.79688,-4.45313 1.28125,-3.48437 1.28125,-3.85937 0,-0.26563 0.1875,-0.48438 0.20312,-0.21875 0.45312,-0.21875 0.0937,-0.10937 0.40625,-0.15625 0.32813,-0.0469 0.51563,-0.0469 l 0.51562,0.51562 q 0.35938,0.32813 0.54688,0.57813 0.1875,0.25 0.1875,0.60937 0,0.57813 -0.20313,1.14063 -0.20312,0.54687 -1.20312,2.625 -0.0625,0.17187 -0.51563,1.15625 -0.3125,0.64062 -0.3125,0.73437 0,0.0937 -0.14062,0.40625 -0.125,0.29688 -0.21875,0.625 -0.25,0.51563 -0.8125,1.8125 -0.5625,1.29688 -0.5625,1.42188 0,0.34375 -0.15625,0.5 -0.0937,0 -0.125,0.15625 -0.0312,0.14062 -0.0312,0.20312 0,0.3125 -0.51563,1.70313 l -0.28125,0.92187 q -0.60937,2.07813 -0.85937,3.04688 -0.23438,0.95312 -0.23438,1.625 0,0.70312 -0.23437,1.15625 -0.21875,0.4375 -0.5,0.57812 z m 13.97729,-12.23437 q 0.29688,0.57812 0.29688,1.3125 0,0.9375 -0.42188,1.53125 -0.3125,0.45312 -0.5625,1.35937 -0.23437,0.89063 -0.45312,2.39063 -0.29688,1.21875 -0.71875,3.75 -0.5,1.34375 -0.625,2.01562 -0.0469,0.1875 -0.3125,0.79688 -0.26563,0.60937 -0.4375,0.95312 -1.4375,2.92188 -3.40625,4.39063 -1.96875,1.46875 -4.01563,1.59375 -0.51562,0.0625 -0.60937,0.0625 -0.28125,0 -0.53125,-0.0937 -0.23438,-0.0937 -0.51563,-0.25 -0.64062,-0.45313 -0.73437,-0.64063 -0.17188,-0.32812 -0.17188,-0.64062 0,-0.48438 0.45313,-0.65625 0.45312,-0.17188 1.03125,-0.17188 0.21875,0.125 0.76562,0.125 0.54688,0 0.64063,-0.125 0.0625,-0.0625 0.375,-0.21875 0.32812,-0.14062 0.73437,-0.23437 1.0625,-0.51563 1.625,-1.03125 0.5625,-0.5 1.23438,-1.625 0.79687,-1.21875 1.04687,-2.20313 0.42188,-1.0625 1,-3.14062 l -0.0312,0.0312 q -0.17187,0.1875 -0.8125,0.5625 -0.625,0.375 -0.76562,0.375 -0.0312,0 -0.0937,0.0312 -0.0625,0.0312 -0.0625,0.125 0,0.125 -0.82813,0.40625 -0.82812,0.26562 -1.0625,0.26562 -0.3125,0 -0.6875,-0.34375 -0.35937,-0.35937 -0.53125,-0.65625 -0.125,-0.28125 -0.46875,-0.57812 -0.39062,-0.375 -0.57812,-0.67188 -0.1875,-0.3125 -0.1875,-0.73437 0,-0.21875 0.0312,-0.34375 0.375,-1.4375 1.40625,-4.42188 1.04687,-3 1.53125,-3.73437 0.125,-0.20313 0.46875,-0.32813 0.35937,-0.125 0.71875,-0.125 0.64062,0 0.85937,0.375 0.39063,0.54688 0.39063,0.9375 0,0.4375 -0.45313,1.3125 -0.125,0.34375 -0.32812,0.76563 -0.1875,0.42187 -0.1875,0.51562 -0.0312,0.125 -0.1875,0.78125 -0.15625,0.65625 -0.28125,0.65625 -0.20313,0.1875 -0.4375,1.32813 -0.23438,1.14062 -0.23438,1.8125 0,0.375 0.0937,0.375 0.85938,0 1.65625,-0.45313 0.79688,-0.46875 1.67188,-1.5 0.125,-0.79687 0.35937,-1.71875 0.25,-0.9375 0.34375,-1.25 0.57813,-1.98437 0.76563,-3.20312 0.1875,-0.45313 0.28125,-0.53125 0.0937,-0.0781 0.42187,-0.0781 0.39063,0 0.84375,0.23438 0.46875,0.21875 0.6875,0.53125 z m 11.99976,13.79687 q -1.25,-0.0312 -1.78125,-0.59375 -0.51563,-0.5625 -0.51563,-1.57812 0,-0.26563 0.0625,-0.84375 0.125,-0.4375 0.1875,-0.89063 0.15625,-0.70312 0.32813,-0.82812 0.0312,-0.0312 0.0625,-0.125 0.0312,-0.10938 -0.0625,-0.17188 0,-0.40625 0.28125,-1.14062 0.0625,-0.23438 0.15625,-0.45313 0.0937,-0.21875 0.14062,-0.28125 -0.0781,-0.0781 -0.0781,-0.20312 0,-0.15625 0.17187,-0.3125 0.1875,-0.15625 0.26563,-0.28125 0.0781,-0.14063 -0.0156,-0.26563 -0.0625,-0.15625 0.60938,-1.4375 0.42187,-0.82812 0.45312,-0.9375 -0.0312,-0.0937 0.45313,-1.17187 0.5,-1.09375 0.79687,-1.64063 0.28125,-0.15625 -0.23437,-0.21875 -0.51563,-0.0625 -1.25,-0.0312 -1.04688,-0.0312 -1.625,-0.10937 -0.57813,-0.0937 -0.9375,-0.28125 -0.21875,-0.21875 -0.21875,-0.57813 0,-0.40625 0.29687,-0.75 0.3125,-0.35937 0.76563,-0.39062 0.82812,-0.0312 2.5625,-0.23438 l 1.625,-0.125 0.0937,-0.54687 q 0.15625,-0.28125 0.29687,-0.48438 0.15625,-0.21875 0.20313,-0.375 0.0469,-0.15625 0.0781,-0.48437 0.0937,-0.125 0.32813,-0.625 0.25,-0.51563 0.3125,-1 0.35937,-0.60938 0.65625,-1.17188 0.29687,-0.5625 0.375,-0.65625 0.5625,-0.85937 1.26562,-0.85937 0.39063,0 0.8125,0.25 0.40625,0.29687 0.625,0.51562 0.23438,0.21875 0.23438,0.60938 0,0.28125 -0.29688,0.82812 -0.125,0.25 -0.375,1.03125 -0.21875,0.28125 -0.48437,0.78125 -0.25,0.5 -0.34375,0.9375 -0.0625,0.15625 -0.20313,0.4375 -0.125,0.26563 -0.17187,0.3125 -0.0469,0.0469 -0.14063,0.0469 0.0312,0.20312 0.125,0.26562 0.0937,0.0625 0.32813,0.0625 0.15625,0 0.39062,-0.0312 0.25,-0.0312 0.59375,-0.0312 0.32813,-0.0312 0.5625,-0.0937 0.23438,-0.0625 0.46875,-0.0625 0.125,0 0.375,0.0937 0.15625,0.1875 0.15625,0.73438 0,0.40625 -0.125,0.78125 -0.125,0.375 -0.3125,0.46875 -0.15625,0.0312 -0.70312,0.1875 -0.54688,0.15625 -1.1875,0.1875 l -1.70313,0.15625 -0.53125,1.09375 q -0.23437,0.39062 -0.39062,0.75 -0.15625,0.34375 -0.21875,0.40625 0,0.15625 -0.26563,0.64062 -0.25,0.48438 -0.375,0.76563 -0.17187,0.45312 -0.39062,0.89062 -0.21875,0.42188 -0.28125,0.54688 -0.0937,0 -0.125,0.0469 -0.0156,0.0469 -0.0156,0.14063 l 0.0781,0.26562 q 0,0.15625 -0.17188,0.3125 -0.125,0.0625 -0.23437,0.23438 -0.10938,0.15625 -0.14063,0.21875 0.0625,0.0625 0.0625,0.15625 0,0.23437 -0.21875,0.57812 0,0.54688 -0.45312,1.625 0.0312,0.57813 0.0625,0.75 0.0312,0.15625 0.15625,0.125 h 0.15625 l 0.70312,-0.0312 q 0.23438,-0.0937 0.10938,0.48438 -0.10938,0.5625 -0.54688,1.75 -0.48437,0.73437 -1.01562,1.29687 -0.51563,0.5625 -0.6875,0.5625 z m 8.84204,-3.14062 q -0.57813,-0.0937 -1.0625,-0.64063 -0.48438,-0.54687 -0.48438,-1.25 0,-0.76562 0.96875,-3.9375 0.15625,-0.82812 0.70313,-2.14062 0.1875,-0.54688 0.21875,-0.64063 0.57812,-1.92187 1.57812,-4.70312 l 0.28125,-0.76563 q 1.125,-3.04687 1.34375,-3.71875 l 0.51563,-1.40625 q 0,-0.25 0.35937,-0.48437 0.375,-0.25 0.625,-0.25 0.51563,0 0.98438,0.67187 0.48437,0.67188 0.51562,1.40625 0,0.39063 -0.15625,0.70313 -0.73437,1.60937 -1.82812,4.96875 -0.46875,1.40625 -0.95313,2.67187 -0.48437,1.26563 -0.76562,1.8125 l -0.35938,0.70313 -0.4375,1.6875 q 0,0.0937 -0.0312,0.17187 -0.0312,0.0625 -0.0312,0.0937 0,0.0937 0.0312,0.0937 l 0.53125,-0.54687 q 0.9375,-0.92188 2.21875,-1.89063 1.04688,-0.73437 1.07813,-0.82812 0.0312,-0.0937 0.26562,-0.29688 l 0.3125,-0.25 q 1.03125,-0.73437 2.21875,-0.73437 0.34375,0 0.82813,0.35937 0.48437,0.34375 0.8125,0.96875 0.34375,0.625 0.34375,1.32813 v 0.25 0.23437 q 0,0.79688 -0.3125,2.09375 -0.29688,1.29688 -0.32813,1.48438 0.15625,0 0.46875,-0.0625 0.48438,0 0.6875,0.0937 0.21875,0.0937 0.375,0.26562 0.15625,0.15625 0.28125,0.1875 0.0625,0.0625 0.0625,0.1875 0,0.73438 -0.79687,1.28125 -0.79688,0.54688 -1.53125,0.60938 -0.89063,0 -1.4375,-0.67188 -0.54688,-0.67187 -0.54688,-1.64062 0,-0.375 0.0312,-0.75 0.0312,-0.39063 0.0625,-0.6875 0.0937,-0.64063 0.0937,-1.01563 0,-0.42187 -0.0781,-0.5625 -0.0781,-0.14062 -0.26563,-0.14062 -0.64062,0 -2.23437,1.17187 -1.57813,1.15625 -3.20313,3.01563 -1.03125,1.5 -1.95312,1.5 z m 17.8548,1.21875 q -1.82813,0 -2.95313,-1.01563 -1.10937,-1.03125 -1.10937,-2.92187 0,-0.73438 0.0937,-1.625 0.0625,-0.78125 0.625,-2.125 0.5625,-1.34375 1.53125,-2.60938 0.96875,-1.28125 2.28125,-1.89062 0.60937,-0.29688 1.03125,-0.40625 0.42187,-0.10938 0.95312,-0.10938 1.3125,0 2.125,0.48438 0.82813,0.46875 1.15625,1.07812 0.34375,0.60938 0.34375,1.0625 v 0.15625 0.0937 q 0,0.39063 -0.45312,1.42188 -0.45313,1.01562 -1.42188,1.92187 -0.96875,0.89063 -2.51562,0.98438 -0.76563,0 -1.48438,-0.14063 -0.71875,-0.15625 -0.90625,-0.15625 -0.48437,0 -0.60937,0.35938 -0.125,0.34375 -0.125,1.1875 0.0937,0.89062 0.48437,1.25 0.40625,0.34375 1.17188,0.34375 0.39062,0 0.89062,-0.21875 0.51563,-0.26563 1.21875,-0.8125 0.39063,-0.3125 0.60938,-0.4375 0.21875,-0.125 0.54687,-0.125 0.28125,0 0.67188,0.39062 0.39062,0.375 0.42187,0.71875 -0.0781,0.39063 -0.71875,1.15625 -0.625,0.76563 -1.65625,1.375 -1.03125,0.60938 -2.20312,0.60938 z m 1.10937,-7.71875 q 0.60938,0 0.84375,-0.125 0.3125,-0.15625 0.96875,-0.8125 0.65625,-0.65625 0.6875,-0.90625 0,-0.32813 -0.28125,-0.53125 -0.26562,-0.20313 -0.73437,-0.28125 h -0.35938 q -0.42187,0 -1.20312,0.70312 -0.78125,0.6875 -1.35938,1.57813 0.0312,0.15625 0.48438,0.26562 0.45312,0.10938 0.95312,0.10938 z m 16.38513,7.90625 q -0.98437,0 -1.85937,-0.4375 -0.85938,-0.45313 -1.375,-1.14063 -0.5,-0.6875 -0.5,-1.35937 0,-0.54688 0.4375,-0.54688 0.29687,0.0625 0.67187,0.35938 0.48438,0.28125 0.8125,0.4375 0.34375,0.15625 0.70313,0.15625 0.53125,0 0.9375,-0.17188 0.40625,-0.17187 0.46875,-0.53125 0,-0.125 -0.14063,-0.54687 -0.125,-0.4375 -0.3125,-0.79688 -0.70312,-1.04687 -0.95312,-2.04687 -0.23438,-1.01563 -0.29688,-2.35938 0,-0.73437 1.0625,-1.75 1.07813,-1.03125 2.5,-1.82812 1.4375,-0.8125 2.32813,-0.90625 1.76562,0 2.10937,0.29687 0.45313,0.34375 0.82813,1.14063 0.39062,0.79687 0.39062,1.34375 0,0.23437 -0.21875,0.57812 -0.20312,0.32813 -0.26562,0.42188 l -0.28125,0.4375 -0.21875,0.48437 q -0.0781,0.15625 -0.21875,0.48438 -0.14063,0.3125 -0.25,0.39062 -0.10938,0.0625 -0.34375,0.0156 -0.53125,0 -0.98438,-0.46875 -0.45312,-0.48438 -0.45312,-1.0625 0,-0.40625 0.28125,-0.82813 0.29687,-0.42187 0.29687,-0.48437 0,-0.125 -0.28125,-0.125 -0.42187,0 -1.29687,0.42187 -0.85938,0.40625 -1.53125,1.09375 -0.67188,0.6875 -0.67188,1.53125 0,0.4375 0.17188,1.03125 0.1875,0.59375 0.40625,0.98438 0.26562,0.25 0.75,1.51562 0.5,1.26563 0.5,2 0,1.125 -0.84375,1.70313 -0.84375,0.5625 -2.35938,0.5625 z m 11.04377,-0.64063 q -0.3125,0.0625 -0.57813,0.0625 -0.40625,0 -0.76562,-0.0781 -0.34375,-0.0781 -0.34375,-0.20313 0,-0.0937 -0.0469,-0.125 -0.0312,-0.0312 -0.125,-0.0312 -0.0937,-0.0937 -0.4375,-0.32812 -0.32812,-0.25 -0.54687,-0.34375 -0.39063,-0.54688 -0.39063,-1.1875 0,-0.60938 0.39063,-1.95313 0.0312,-1.25 0.90625,-2.79687 0.875,-1.54688 2.125,-2.875 1.25,-1.32813 2.25,-1.90625 0.46875,-0.48438 1.34375,-0.48438 0.67187,0 1.1875,0.28125 0.40625,0.26563 1.03125,1.09375 0.625,0.82813 0.9375,1.5 0.0781,0.23438 0.1875,0.625 0.10937,0.375 0.10937,0.92188 l -0.0625,0.89062 q -0.39062,1.3125 -0.39062,2.35938 -0.0625,0.45312 0.0625,0.73437 0.14062,0.26563 0.35937,0.26563 0.15625,0 0.34375,-0.0937 0.20313,-0.0937 0.45313,-0.0937 0.42187,0 0.70312,0.25 0.29688,0.25 0.29688,0.64062 0,0.3125 -0.17188,0.60938 -0.28125,0.64062 -1.0625,1.15625 -0.78125,0.5 -1.45312,0.5 -0.54688,0 -1.04688,-0.5625 -0.5,-0.57813 -0.78125,-1.51563 l -0.1875,-0.73437 -1.15625,1.0625 -0.73437,0.73437 q -0.51563,0.51563 -0.57813,0.60938 -0.15625,0.25 -0.5625,0.48437 -0.40625,0.21875 -1.26562,0.53125 z m 0.85937,-3.0625 q 0.57813,-0.32812 0.96875,-0.79687 0.40625,-0.48438 1.01563,-1.34375 0.14062,-0.20313 0.26562,-0.40625 0.125,-0.20313 0.3125,-0.4375 0.54688,-0.73438 0.78125,-1.17188 0.25,-0.45312 0.375,-0.96875 -0.125,-0.42187 -0.375,-0.78125 -0.23437,-0.375 -0.42187,-0.40625 -0.90625,0 -2.4375,2.10938 -0.15625,0.23437 -0.5,0.8125 -0.32813,0.5625 -0.42188,0.76562 -0.17187,0.21875 -0.4375,1.07813 -0.26562,0.84375 -0.26562,1.09375 0,0.70312 0.40625,0.70312 0.32812,0 0.73437,-0.25 z m 25.32318,0.92188 q 0.21875,0.32812 0.21875,0.60937 0,0.32813 -0.23437,0.57813 -0.21875,0.25 -0.375,0.25 -0.76563,0 -1.71875,-0.35938 -0.9375,-0.375 -0.9375,-1.07812 v -1.98438 q 0,-1.28125 -0.21875,-1.46875 -0.48438,-0.25 -1.28125,0.71875 -0.79688,0.96875 -1.95313,2.89063 -0.45312,0.70312 -0.78125,1.01562 -0.3125,0.29688 -0.5625,0.29688 -0.21875,0 -0.39062,-0.125 -0.98438,-0.28125 -0.98438,-1.125 0,-0.0937 0.0625,-0.40625 0.28125,-1 0.60938,-2.20313 0.34375,-1.21875 0.54687,-2.09375 -0.0937,-0.1875 -0.32812,-0.1875 -0.3125,0 -0.73438,0.32813 -0.125,0.0937 -0.46875,0.29687 -0.32812,0.20313 -0.67187,0.54688 -0.32813,0.32812 -0.6875,0.90625 -0.125,0.23437 -0.4375,0.67187 -1.60938,2.4375 -1.95313,2.75 -0.17187,0.45313 -0.3125,0.64063 -0.14062,0.17187 -0.32812,0.17187 -0.125,0 -0.32813,-0.10937 -0.60937,-0.28125 -0.98437,-0.8125 -0.35938,-0.53125 -0.35938,-1.26563 0,-0.46875 0.28125,-1.90625 0.29688,-1.45312 0.65625,-2.70312 0.25,-0.5 0.39063,-1.0625 0.14062,-0.5625 0.14062,-0.78125 0.10938,-0.60938 0.39063,-1.64063 0.28125,-1.04687 0.42187,-1.15625 0.1875,-0.1875 0.70313,-0.1875 0.34375,0 0.65625,0.14063 0.32812,0.125 0.45312,0.375 0.42188,0.48437 0.42188,1.3125 0,0.76562 -0.3125,1.625 -0.32813,0.73437 -0.8125,2.15625 0,0 0.0625,-0.0781 0.78125,-1.07812 1.98437,-2.42187 0.57813,-0.64063 1.29688,-0.98438 0.71875,-0.35937 1.17187,-0.35937 0.76563,0.125 1.29688,0.75 0.53125,0.625 0.65625,1.70312 0,0.39063 -0.0625,0.95313 -0.0625,0.5625 -0.0937,0.71875 0.125,-0.15625 0.65625,-0.78125 0.53125,-0.625 1.29687,-1.29688 0.70313,-0.54687 1.3125,-0.54687 0.5,0 0.89063,0.42187 0.70312,0.40625 1.09375,1.34375 0.40625,0.92188 0.40625,2.10938 0,0.40625 -0.0312,0.64062 -0.0312,0.21875 -0.0312,0.64063 0,0.4375 0.0937,0.84375 0.10937,0.40625 0.20312,0.71875 z m 4.44238,2.3125 q -1.0625,-0.42188 -1.5625,-1.29688 -0.48437,-0.89062 -0.48437,-2.29687 0,-0.3125 0.0625,-1.15625 0.0312,-0.25 0.0312,-0.59375 0,-0.54688 -0.125,-0.8125 -0.125,-0.28125 -0.45313,-0.57813 -0.0937,-0.0937 -0.45312,-0.40625 -0.34375,-0.32812 -0.3125,-0.48437 v -0.0937 q 0,-0.21875 0.10937,-0.48438 0.10938,-0.28125 0.21875,-0.34375 0.0937,-0.0312 0.375,-0.1875 0.29688,-0.17187 0.45313,-0.10937 0.125,0 0.39062,0.17187 0.28125,0.15625 0.28125,0.28125 0.1875,0.39063 0.98438,-1.25 0.45312,-0.67187 1.21875,-1.32812 0.76562,-0.65625 1.40625,-1 0.23437,-0.10938 1.01562,-0.28125 0.78125,-0.17188 1.23438,-0.17188 0.5,0 1.1875,0.42188 0.6875,0.40625 1.14062,0.89062 0.67188,0.48438 0.67188,1.82813 0,0.79687 -0.25,1.375 -0.125,0.46875 -1.26563,1.45312 -1.14062,0.96875 -1.84375,1.26563 -0.70312,0.15625 -1.5625,0.15625 -0.875,0 -1.1875,-0.15625 -0.21875,-0.125 -0.375,0.10937 -0.14062,0.23438 -0.23437,1.07813 l -0.0312,0.3125 q 0,0.35937 0.15625,0.67187 0.23437,0.39063 0.42187,0.54688 0.1875,0.15625 0.57813,0.15625 l 0.28125,-0.0312 q 0.35937,-0.0312 0.75,-0.34375 0.40625,-0.32813 0.95312,-0.875 0.57813,-0.54688 0.67188,-0.64063 0.15625,-0.3125 0.85937,-0.3125 0.28125,0 0.73438,0.0937 0.32812,0.25 0.32812,0.64063 0,0.54687 -0.51562,1.46875 -0.15625,0.32812 -0.98438,1.09375 -0.8125,0.76562 -1.29687,1.04687 -0.85938,0.48438 -1.90625,0.48438 -0.8125,0 -1.67188,-0.3125 z m 2.375,-7.20313 q 0.76563,-0.20312 1.1875,-0.46875 0.4375,-0.26562 1.01563,-1.07812 0.25,-0.76563 0.0937,-0.92188 -0.1875,-0.1875 -0.45313,-0.1875 -0.3125,0 -0.76562,0.34375 -0.3125,0.0937 -0.875,0.59375 -0.5625,0.5 -0.95313,0.96875 -0.375,0.45313 -0.21875,0.54688 -0.0625,0.10937 0.3125,0.15625 0.39063,0.0469 0.65625,0.0469 z"
id="path4094"
inkscape:connector-curvature="0"
style="fill:#000000;fill-rule:nonzero" />
<path
d="M 427.809,138.3307 H 553.08723 V 253.13125 H 427.809 Z"
id="path4100"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="M 379.751,138.43701 H 505.04235 V 253.24016 H 379.751 Z"
id="path4109"
inkscape:connector-curvature="0"
style="fill:#00fdc8;fill-rule:evenodd" />
<path
d="M 379.751,138.43701 H 505.04235 V 253.24016 H 379.751 Z"
id="path4111"
inkscape:connector-curvature="0"
style="fill-rule:evenodd;stroke:#666666;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
<path
d="m 299.38583,404.37924 h 70.36065 v -6.68504 l 8.40525,13.37008 -8.40525,13.37007 v -6.68502 h -70.36065 z"
id="path4071-3"
inkscape:connector-curvature="0"
style="fill:#7c7ce0;fill-rule:evenodd;stroke-width:0.79288208" />
<path
d="m 506.71874,190.09462 h 70.36065 v -6.68504 l 8.40525,13.37008 -8.40525,13.37007 v -6.68502 h -70.36065 z"
id="path4071-6"
inkscape:connector-curvature="0"
style="fill:#7c7ce0;fill-rule:evenodd;stroke-width:0.79288208" />
<path
d="m 506.71875,404.41666 h 70.36065 v -6.68504 l 8.40525,13.37008 -8.40525,13.37007 v -6.68502 h -70.36065 z"
id="path4071-6-7"
inkscape:connector-curvature="0"
style="fill:#7c7ce0;fill-rule:evenodd;stroke-width:0.79288208" />
<flowRoot
xml:space="preserve"
id="flowRoot4272"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:26.66666603px;line-height:1.25;font-family:'Comic Sans MS';-inkscape-font-specification:'Comic Sans MS';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none"><flowRegion
id="flowRegion4274"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:26.66666603px;font-family:'Comic Sans MS';-inkscape-font-specification:'Comic Sans MS';text-align:center;text-anchor:middle"><rect
id="rect4276"
width="125.97456"
height="90.782555"
x="379.06781"
y="162.45763"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:26.66666603px;font-family:'Comic Sans MS';-inkscape-font-specification:'Comic Sans MS';text-align:center;text-anchor:middle" /></flowRegion><flowPara
id="flowPara4278">Some process</flowPara></flowRoot> <g
transform="matrix(0.45555722,0,0,0.45555774,379.751,138.43701)"
id="g4082-1"
inkscape:transform-center-x="-34.322034"
inkscape:transform-center-y="102.9661">
<clipPath
id="clipPath4292">
<path
d="M 0,0 H 275 V 252 H 0 Z"
id="path4290"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
<image
clip-path="url(#p.3-4)"
width="275"
height="252"
x="0"
y="0"
preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/magic.gif"
id="image4080-3"
style="fill:#000000" />
</g>
<rect
style="fill:#ffffff"
id="rect148"
width="28.983051"
height="45.762711"
x="212.03391"
y="388.22034" />
</g>
</svg>
</center></p>
<p>Sounds interesting? Excellent! Start with these two articles.</p>
<ul>
<li><a href="differential-privacy-awesomeness.html">Why differential privacy is awesome</a> presents a high-level,
non-technical description of the definition. Then, it explains what makes it
so great, and why people are so excited about it.</li>
<li><a href="differential-privacy-in-more-detail.html">Differential privacy in (a bit) more detail</a> introduces the formal
definition. It uses very little math, and explains every single concept needed
to understand it.</li>
</ul>
<p>Then, this blog post series splits in two branches. You can read one or the
other in any order, depending on what you're most interested in.</p>
<p>The first branch is about the <em>how</em>: what techniques can you use to achieve
differential privacy? It's a <em>little bit</em> technical, though I still keep it as
simple as I can. If that doesn't sound interesting, feel free to skip over and
go directly to the next section!</p>
<ul>
<li><a href="differential-privacy-in-practice.html">Differential privacy in practice (easy version)</a> explains how to
make simple statistics differentially private: counts, sums, histograms… In
many cases, you don't need much more!</li>
<li><a href="almost-differential-privacy.html">Almost differential privacy</a> describes how to publish private
histograms without knowing the categories in advance. It also introduces an
important variant: <span class="math">\((\varepsilon,\delta)\)</span>-differential privacy.</li>
<li><a href="privacy-loss-random-variable.html">The privacy loss random variable</a> introduces this central concept in
differential privacy. Then, it uses it to explain the real meaning of <span class="math">\(\delta\)</span>
in <span class="math">\((\varepsilon,\delta)\)</span>-differential privacy.</li>
<li><a href="gaussian-noise.html">The magic of Gaussian noise</a> introduces Gaussian noise and its
shiny properties. In particular, it explains why Gaussian noise is so useful
when releasing many statistics.</li>
<li><a href="more-useful-results-dp.html">Getting more useful results with differential privacy</a> presents five
simple techniques to improve the utility of your anonymized data. If you're
rolling out your first differential privacy pipeline and it's not performing
well, this is the article for you!</li>
<li><a href="renyi-dp-zero-concentrated-dp.html">Averaging risk: Rényi DP & zero-concentrated DP</a> introduces two
variants of differential privacy. By averaging the privacy loss, these notions
allow for much better privacy analysis in a number of common use cases.</li>
<li><a href="choosing-things-privately.html">Choosing things privately with the exponential mechanism</a> explains
a fundamental building block of differential privacy: a mechanism to privately
select the best option among many.</li>
</ul>
<p>The second branch of these series is about the <em>why</em>. In which contexts can
differential privacy be used? Why do organizations decide to adopt it? What
policy questions does it raise? The articles in this branch are accessible to
non-technical folks, and are all self-contained. You can read them in any order
you like!</p>
<ul>
<li><a href="local-global-differential-privacy.html">Local vs. central differential privacy</a> presents the two main <em>models</em>
of differential privacy, depending on who the attacker is. <em>Publishing</em>
provably anonymized data is very different from <em>collecting</em> it, and this post
explains the difference.</li>
<li><a href="why-not-differential-privacy.html">Why not differential privacy?</a> explores what it means for an
algorithm to <em>not</em> be differentially private. Don't understand why people
insist on using differential privacy, even when it seems overkill? This is the
resource for you.</li>
<li><a href="us-census-reconstruction-attack.html">Demystifying the US Census Bureau's reconstruction attack</a>
explains why the government agency decided to use differential privacy. They
ran an attack on the data they published in 2010, and the results are quite
scary.</li>
<li><a href="noisy-data.html">Don't worry, your data's noisy</a> makes the case that <em>all</em> data is
noisy, even without differential privacy. In fact, the noise used in
differential privacy is the best kind of noise: the one you can quantify.</li>
<li><a href="litmus-test-differential-privacy.html">Is differential privacy the right fit for your problem?</a> gives a
simple litmus test to quickly get an idea of whether a given use case is
well-suited to using differential privacy.</li>
<li><a href="trustworthy-anonymization.html">What anonymization techniques can you trust?</a> lists some common legacy
techniques used to anonymize data, explains how they fail, and draws some
lessons from these historical failures.</li>
<li><a href="privacy-enhancing-technologies.html">Mapping privacy-enhancing technologies to your use cases</a> situates
differential privacy in the wider context of privacy-enhancing technologies,
explaining which kind of data privacy challenge each one addresses.</li>
</ul>
<p>Finally, one article lists the <a href="real-world-differential-privacy.html">known real-world deployments of DP</a>,
along with their privacy parameters.</p>
<p>This series isn't finished. I have a list of future articles I'd like to write…
and I'm adding new ideas to this list faster than I'm writing blog posts! If
you're looking for further things to read on differential privacy, you can do
two things.</p>
<ul>
<li>You can check out this <a href="differential-privacy-reading-list.html">reading list</a> I curated. I particularly recommend it
you're looking for more formal content: textbooks with mathematical proofs,
scientific papers, etc.</li>
<li>You can follow me on <a href="https://hachyderm.io/@tedted">Mastodon</a> or subscribe to this blog's <a href="feed.xml">RSS feed</a> to
keep updated about future posts.</li>
</ul>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Joining Tumult Labs2021-08-05T00:00:00+02:002021-08-05T00:00:00+02:00Damien Desfontainestag:desfontain.es,2021-08-05:/privacy/joining-tumult-labs.html<p>I'm joining Tumult Labs, a startup focused on differential privacy. Here's why I'm excited about it!</p><p><strong>I'm</strong> joining <a href="https://www.tmlt.io">Tumult Labs</a>, a startup focused on making
<a href="differential-privacy-awesomeness.html">differential privacy</a> ubiquitous. I'm incredibly excited about taking DP to
the next stage, and I think Tumult Labs is a fantastic place to do that. Let me
tell you why!</p>
<h1 id="why-differential-privacy">Why differential privacy?</h1>
<p><a href="differential-privacy-awesomeness.html">Differential privacy</a> (DP) makes it possible to safely publish statistics
about sensitive data. Today, using DP in practice isn't very easy, but potential
applications are everywhere. I'm hopeful that by making this tech more usable
and more useful, we can make progress on three fronts.</p>
<h4 id="making-existing-practices-safer">Making existing practices safer</h4>
<p>Today, many organizations compute statistics about people, and share it with
third parties. Government agencies publish information about their citizens.
Online platforms give content creators data about their audience. Hospitals
share patient data with researchers. And of course, a myriad of commercial
applications exist too.</p>
<p>All this data is often more revealing than people expect: anonymization is
difficult to get right, and easy to get wrong. Problems happen <a href="https://www.smh.com.au/technology/australians-health-records-unwittingly-exposed-20171218-p4yxt2.html">all</a>
<a href="https://www.wired.com/2007/12/why-anonymous-data-sometimes-isnt/">the</a> <a href="https://www.canberratimes.com.au/story/6330850/vic-transport-data-not-anonymous-enough/">time</a>. Worse, these problems are especially harmful to
folks in <a href="https://www.vox.com/recode/22587248/grindr-app-location-data-outed-priest-jeffrey-burrill-pillar-data-harvesting">marginalized populations</a>. Only releasing aggregate data,
or using clever-looking mitigations, isn't enough. The US Census <a href="us-census-reconstruction-attack.html">reconstruction
attack</a> made that very clear.</p>
<p>Worse, these are only the problems we hear about. Many organizations generate
and archive statistical data, without ever sharing it with outsiders. What if
they misclassify this data as "anonymized", even though reidentification is
possible? This can have painful compliance implications, even though it might
not make headlines.</p>
<p>Differential privacy is the only tool we know that prevents these problems. The
math works: DP provides <em>provable</em> guarantees. The main obstacle to adoption is
usability. So let's work on that, and make all these practices much safer!</p>
<h4 id="enabling-new-use-cases">Enabling new use cases</h4>
<p>These existing practices are only the tip of the iceberg. How many applications
never see the light of day, for privacy and compliance reasons? Governments want
to do more <a href="https://en.wikipedia.org/wiki/Open_data">open data</a>, but don't want to breach their constituents'
privacy. Organizations want to share data with researchers, without risking
privacy incidents. Valuable data gets deleted, even though it could have been
useful to share it with others, or archive some of it.</p>
<p>Robust and accessible anonymization tooling will enable many such use cases. It
will lower the cost of both compliance and respectful privacy practices. Today,
DP is still a shiny new concept that requires considerable investment to use:
only large organizations can afford it. In a few years, we can turn DP into a
standard industry best practice, like encryption at rest or audit trails. I
can't wait for us to get there!</p>
<h4 id="improving-incentive-structures">Improving incentive structures</h4>
<p>What happens once it gets dead simple to use provable anonymization techniques?
Enabling the use cases I mentioned before is one thing. But there might also be
rippling beneficial effects, much more impactful in the long run.</p>
<p>Here's an example. Today, large companies can see value in sharing data with
researchers. But the choice is theirs. What if they don't see the point? Or
worse, what if they're afraid of what the research might uncover? Nothing stops
them from keeping the data secret. Pushing these companies to be more
transparent about their impact is difficult: privacy can be a convenient excuse
to keep damaging information secret.</p>
<p>Once differential privacy is the new standard, and using it is easy, this excuse
goes away. It will be easier to demand openness from companies holding massive
amounts of data. This could happen, for example, through legislation. Today,
regulators who want to mandate such transparency face a difficult task: how to
do so without creating requirements that conflict with data protection
regulations? Differential privacy could change the equation.</p>
<p>And legislation isn't the only path: changing <em>cultural norms</em> can have a
massive impact as well. The easier it is to share data, the more organizations
will. Over time, the perception of what is and isn't legitimate to keep secret
will shift. And such a global shift towards openness can create a lot of
pressure!</p>
<p>So that's why I'm excited about differential privacy. Now, why do I think
Tumult Labs is a great place for making it successful?</p>
<h1 id="why-tumult-labs">Why Tumult Labs?</h1>
<p>Tumult Labs was founded by three professors, who have been working on DP for
more than a decade. They worked on <a href="https://scholar.google.com/scholar?as_q=&as_epq=%22differential+privacy%22&as_oq=%22ashin+machanavajjhala%22+%22michael+hay%22+%22gerome+miklau%22">foundational theory</a> <em>and</em> on DP's
<a href="http://www.cse.psu.edu/~duk17/papers/PrivacyOnTheMap.pdf">first-ever real-world deployment</a>. Now, they're advising the US
Census Bureau on some of their <a href="https://arxiv.org/pdf/2107.10659.pdf">most complex challenges</a>.</p>
<p>Their work on real-world deployments led them to a realization I'm quite
familiar with: if we want to reach wide adoption of DP, we have to invest in
building excellent tooling. So they've been assembling a <a href="https://www.tmlt.io/company">talented
team</a> to do exactly that: build what their clients
need into a single general-purpose platform. And I was very impressed by what
they <a href="https://users.cs.duke.edu/~ashwin/pubs/Ektelo-SIGMOD2018.pdf">published</a> about this work! It shows both deep technical insight,
and a focus on building things in a modular and reusable way. I can't wait to
dig into this tech some more, and learn from the folks behind it.</p>
<p>Finally, I'm also eager to tackle the personal challenges that await. Being a
part of a small startup should be a much-needed change of pace. Helping external
clients will be new and pretty exciting. I'll also be Tumult's first employee in
Europe, but not the last: I'm thrilled about the idea of growing the team there.
Speaking of which, if the mission resonates with you, check out our <a href="https://www.tmlt.io/careers">careers</a>
page, or drop me a line directly!</p>Don't worry, your data's noisy2021-07-27T00:00:00+02:002021-07-27T00:00:00+02:00Damien Desfontainestag:desfontain.es,2021-07-27:/privacy/noisy-data.html<p>Your data was already noisy before I got a chance to add noise to it! Here's why you shouldn't panic, and also what you should do about it.</p><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p></small></p>
<hr>
<p><span class='lettrine'>H</span><strong>ere</strong> is a cold, hard, inescapable truth: your
data has noise in it. No, we're not talking about differential privacy (yet)!
Nobody added random numbers to your statistics so far. But still, your data is
noisy. It's wrong. It's not 100% accurate. It's uncertain. Worse, there are <em>two
kinds</em> of uncertainty.</p>
<h1 id="two-kinds-of-uncertainty">Two kinds of uncertainty</h1>
<p>Let's start with the first kind. Here's a diagram.</p>
<p><center>
<img alt="The picture is split in two. On the left, a point is labeled "What you think
your data looks like". On the right, the same point with a confidence interval
(a line that ends with small perpendicular lines) is labeled "What your data
actually looks like"." src="https://desfontain.es/privacy/images/point-vs-confidence-interval.svg">
</center></p>
<p>The point on the left is how most people think about statistics. Precise numbers
that corresponds to exact truths. Take, for example, a database with eye color
information in it. Here is a statistic: « there are 4217 people with brown eyes
in this database ». What did you learn about the world when reading this
statistic? One answer could be « well, I've learned that the dataset contains
4217 people with brown eyes ». That sounds reasonable enough. Right?</p>
<p>Wrong.</p>
<p>At best, you've learned that the number of people with brown eyes in the
database is <em>around</em> 4217. It might be the best estimate you've got. But is this
statistic 100% accurate? Would you bet money on it being 100% accurate? What
would it take for you to be willing to take such a bet?</p>
<p>I can hear your objections already. « Whoa, wait a second. What does 100%
accurate means? How is eye color defined exactly? How do we double-check the
number? And how was this statistic generated, anyway? Did each person answer a
question about their own eye color? Or did someone else do the classifying? Is
eye color information available for 100% of the people in the database? » And so
on. You probably have many other valid questions. Behind those questions lies
the deep, uncomfortable truth: this statistic is almost certainly noisy.</p>
<p>At best, you need to add error bars if you want to represent this statistic in
an honest way. This is what happens on the right side of the previous picture.
We've got an exact number, represented by a point, and a <em>confidence interval</em>
(or <em>error bars</em>) around this point. This statistic no longer says « there are
exactly 4217 people with brown eyes in this database ». Instead, it says
something more complex, but more accurate. « With 95% certainty, there are
between 4209 and 4226 browned-eyed people in the database. The best estimate
we've got is 4217. »</p>
<p>Wait a second.</p>
<p><center>
<img alt="A scene from Futurama, where Bender picks up a cigar and smokes it, while
Hermes and Zoidberg look at the scene from the background. Hermes' line appears
in a subtitle: "That just raises further
questions!"" src="https://desfontain.es/privacy/images/further-questions.gif">
</center></p>
<p>Right. How was this confidence interval computed? What do we mean by 95%
certainty? What is the uncertainty capturing? Did we miss some sources of
uncertainty? Is there uncertainty in this uncertainty estimate<sup id="fnref:uncertainty"><a class="footnote-ref" href="#fn:uncertainty">1</a></sup>?</p>
<p>This leads me to my second point, even more devastating than the first. Your
data is noisy, and also, you probably don't even know <em>how much noise</em> is in it.
The statistics you have are best guesses, not much more.</p>
<p><center>
<img alt="A diagram similar to the previous one, split in two. On the left, the same
confidence interval as above is labeled "What you wish your data looked like".
On the right, the confidence interval has been replaced by a dotted-line
representing unknown uncertainty, and labeled "What your data actually looks
like"." src="https://desfontain.es/privacy/images/confidence-interval-fuzzy-uncertainty.svg">
</center></p>
<p>The figure on the right represents this unknown uncertainty. You suspect that
the number you have isn't too far removed from the actual number. But you can't
quantify how far.</p>
<p>Let's take a closer look at both kinds of uncertainty. Quantifiable
uncertainties can take several forms.</p>
<ul>
<li>Your data might be a uniform sample of a larger population. In this case, you
can calculate the
<a href="https://en.wikipedia.org/wiki/Sampling_error">sampling error</a> of your
statistics.</li>
<li>A large-scale data collection process might miss a few events. This happens,
for example, when servers crash, or network problems occur. But you might be
able to estimate how often that happens, and say the uncertainty is below e.g.
1%.</li>
<li>The accuracy of certain algorithms can be measured. For example, you train
machine learning algorithms on a training set, and evaluate them on a test
set.</li>
<li>If humans are labelling data by hand, you can have several analysts label
the same data. This way, you can get an idea of how often they agree, and
estimate uncertainty this way.</li>
</ul>
<p>After quantifying the uncertainty, you can take it into account in the analyses
you run. For example, you can propagate the error bars. Or only return results
if they are very likely to still be valid <em>despite</em> the
sources of error.</p>
<p>Meanwhile, unquantifiable uncertainties can also come from several places.</p>
<ul>
<li>In practice, statistical sampling is often not uniform: your statistics might
suffer from <a href="https://en.wikipedia.org/wiki/Selection_bias">selection bias</a> or
<a href="https://en.wikipedia.org/wiki/Survivorship_bias">survivorship bias</a>. You can
guesstimate the effect of those on your data, but it's hard to be 100% sure.</li>
<li>When you ask people questions, their answers might not be accurate. This is
called <a href="https://en.wikipedia.org/wiki/Reporting_bias">reporting bias</a>. There
are many reasons why it can happen, and the overall impact is also difficult
to estimate.</li>
<li>People sometimes dig into data until they find something interesting to
report. This practice, called
<a href="https://en.wikipedia.org/wiki/Data_dredging">data dredging</a>, generates
good-looking, but completely meaningless results. This is made worse by the
fact that positive results are
<a href="https://en.wikipedia.org/wiki/Publication_bias">more likely to get published</a>.
Both phenomena lead to real problems in practice! They create inherent
uncertainty in the results reported in the scientific literature. And again,
it's hard to quantify this uncertainty.</li>
</ul>
<p>Sometimes, the situation is even worse than that. People might use ad hoc
protection techniques that add some fuzziness to the data, and <em>not tell you
about it</em>. Or they might give you an idea, but no details. This is what the US
Census Bureau did for their 2000 and 2010 releases: they randomly swapped
records, but didn't publish any details about how that procedure worked. Sadly,
it <a href="us-census-reconstruction-attack.html">didn't even succeed</a> in protecting the data. But it did make
the entire data noisy in ways nobody could find out, nor take into account.</p>
<p>Often, both quantifiable and unquantifiable effects end up affecting your data.
So you end up with something like this.</p>
<p><center>
<img alt="A diagram combining a confidence interval (in blue) and dotted-line unknown
uncertainty (in brown), around the same point. It's label "What your data ends
up looking like"." src="https://desfontain.es/privacy/images/combined-uncertainties.svg">
</center></p>
<p>The quantifiable uncertainty, for which you can draw confidence intervals, is in
blue. But there is still some unquantifiable uncertainty, here in brown: you
should take the whole thing, including the error bars, with a grain of salt.</p>
<p>You probably see what's coming next. What if we add noise to the statistics, to
get <a href="differential-privacy-awesomeness.html">differential privacy</a> guarantees?</p>
<h1 id="adding-differential-privacy-to-the-mix">Adding differential privacy to the mix</h1>
<p>Differential privacy is typically obtained by adding <em>noise</em> to statistics. We
pick a number at random according to some distribution, and add it to the
statistics. This distribution isn't secret, only the random number is. So this
is the nice kind of uncertainty: the one we can quantify. If you already have
some quantifiable uncertainty, you can combine both, and get a single confidence
interval. And of course, the whole thing might still be uncertain in ways we
can't quite estimate.</p>
<p><center>
<img alt="Another diagram split in two. On the left, the same picture as the previous
one (with both a confidence interval and unknown uncertainty) is labeled "Before
DP". On the right, the same picture is duplicated twice, with an "approximately
equal" sign between both duplicates; one duplicate has an additional confidence
interval in red, the other has both confidence intervals combined into a purple,
larger one. The right picture is labeled "After
DP"." src="https://desfontain.es/privacy/images/uncertainty-before-after-dp.svg">
</center></p>
<p>We didn't change the situation much. We only made the confidence interval a
little bit wider. The quantifiable uncertainty increased a little, and we could
quantify how much.</p>
<p>Some DP algorithms are more complex, and add noise to data in more creative
ways. Computing the uncertainty for these algorithms can be difficult. But most
building blocks can be analyzed to find out confidence intervals. This might be
a bit of a pain, so in an ideal world, you don't have to do this by hand: the
tooling you use does it for you.</p>
<p>Sometimes, though, you get the not-so-nice kind of uncertainty. For example, you
might need to <a href="differential-privacy-in-practice.html#counting-things">limit the contribution</a> of each person in your database.
This can take several forms, like <em>clamping</em> or <em>subsampling</em>. This creates
additional error, or uncertainty, on top of the noise itself. But this one is
hard to quantify: in theory, someone could contribute a billion things, and we
would only count 5 of them. In that case, the error due to this single outlier
would be huge. </p>
<p>If you want to quantify this effect, you have two options. The people publishing
the data can tell you the magnitude of data loss due to clamping. If that didn't
happen, you can make some reasonable assumptions on these outliers: how many
there are, and how much data was dropped. In both cases, note this clamping is
often <em>positive</em> for data quality: robust statistical analyses shouldn't be
sensitive to huge outliers.</p>
<h1 id="conclusion">Conclusion</h1>
<p>Differential privacy's effect on data isn't as world-ending as you might think.
Your data didn't go from perfect to noisy. It was noisy all along! DP only made
it a little more so. And you can quantify the effect of this new noise, which is
nice. (At least, nicer than some of the uncertainty sources that were already
there.)</p>
<p>This might come off as a surprise. You might have been using your data as if it
was a perfect source of absolute truth. Not the most scientific approach, but
maybe that was good enough for your application. In that case… you can probably
continue doing the same with DP data? If the people who designed the DP
process did a decent job, the statistics shouldn't be too far from the real
data. Unless you were looking at very small statistics, of course. But in that
case, you were looking at pure statistical noise in the first place.</p>
<p>If you were already taking uncertainty into account, DP doesn't change much. It
only adds a new, quantifiable element to this uncertainty. With one caveat: you
need to know exactly what was done to the data. Privacy parameters aren't enough
for you to guess. You need to know the shape and magnitude of the noise, and all
other ways in which the data was altered.</p>
<p>This also means that DP practitioners should give you this information. People
creating and implementing DP algorithms share part of this responsibility, too.
Novel algorithms should come with a way to calculate arbitrary confidence
intervals. Tools should return uncertainty information along with the output:
nobody wants to compute it by hand. </p>
<p>In a few words: transparency is key. It brings trust. It makes the data more
useful. And it can show that the accuracy impact of DP is smaller than people
might think!</p>
<hr>
<p>For more musings on differential privacy, head over to the <a href="friendly-intro-to-differential-privacy.html">table of
contents</a> of this blog post series.</p>
<hr>
<p><small>I'm thankful to Cynthia Dwork and danah boyd for their helpful feedback
on drafts of this blog post.</small></p>
<div class="footnote">
<hr>
<ol>
<li id="fn:uncertainty">
<p>This is not a silly question! And it can go further. I have seen
physicists calculating confidence intervals for confidence interval bounds
<em>for confidence interval bounds</em>. It is a real thing that very serious
scientists sometimes do. <a class="footnote-backref" href="#fnref:uncertainty" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Getting more useful results with differential privacy2021-06-16T00:00:00+02:002021-07-14T00:00:00+02:00Desfontaines, Damien and Simmons-Marengo, Danieltag:desfontain.es,2021-06-16:/privacy/more-useful-results-dp.html<p>A few generic pieces of advice on how to get better utility out of your differentially private aggregations.</p><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p>This post was co-written by Daniel Simmons-Marengo and myself.
</small></p>
<hr>
<p><span class='lettrine'>S</span><strong>o</strong> you decided to use differential privacy
(DP) to publish or share some statistical data. You ran your first <a href="https://github.com/google/differential-privacy/tree/main/privacy-on-beam">pipeline</a> or
<a href="https://github.com/google/differential-privacy/tree/main/examples/zetasql">query</a><sup id="fnref:osstools"><a class="footnote-ref" href="#fn:osstools">1</a></sup>, all excited, but then… You're getting useless
results. Maybe all your data seems to have disappeared. Or the statistics look
very unreliable: the noise completely drowns out the useful signal. Don't lose
hope! This situation is common the first time people try to use differential
privacy. And chances are that you can fix it with a few simple changes.</p>
<p>In this post, we'll cover five basic strategies to improve the utility of your
anonymized data. These are far from the only tricks you can use, but they're a
good place to start. And none of them requires you to sacrifice any privacy
guarantees.</p>
<h1 id="aggregate-as-much-data-as-coarsely-as-possible">Aggregate as much data, as coarsely, as possible</h1>
<p>DP algorithms produce better results when run over more data. Remember: the
noise they introduce is proportional to the contribution of a single person. It
doesn't depend on the size of the input data. So, the more people you have in a
statistic, the smaller the <em>relative</em> noise will be. Individual contributions
will "vanish into the crowd", and so will the added uncertainty.</p>
<p>Increasing the total amount of input data will improve utility, but you may not
be able to get more data. Luckily, there are other ways to take advantage of
this property. What matters is the amount of data that contributes to each
<em>statistic</em> you produce. In other words, the finer you slice and dice your data,
the worse your utility will be. If you can, slice the data into coarser
partitions. For example, calculate weekly statistics rather than daily
statistics. Or aggregate your data per-country rather than per-city. You get the
idea.</p>
<p><center>
<img alt="Visual aid showing the data represented as a square cut into four big pieces
on the left, and the same square cut into sixteen pieces on the right, with an
arrow going from left to right between both
diagrams." src="https://desfontain.es/privacy/images/aggregate-more-coarsely.svg">
</center></p>
<p>Another common trick is to slice by fewer dimensions at the same time. Suppose
that your query calculates the number of visitors by country and language. Do
you need the combination of both dimensions? Many combinations of country and
language are rare, and will only have a few visitors. Instead, calculating them
separately might work better: you will get more users in each statistic, so the
overall impact of the noise might be lower.</p>
<h1 id="minimize-the-number-of-aggregations">Minimize the number of aggregations</h1>
<p>DP bounds the total amount you can learn about any individual from a series of
data releases. Every statistic you calculate reveals <em>some</em> information about
individuals in the data. To bound the global privacy cost of a data release, you
have to set a <a href="differential-privacy-in-practice.html#multiple-statistics"><em>privacy budget</em></a>. This is the total <span class="math">\(\varepsilon\)</span> and
<span class="math">\(\delta\)</span> cost of your set of calculations.</p>
<p>Each statistic spends part of this privacy budget. So if you have a fixed
privacy budget, and you want to calculate more statistics, each one must reveal
less. That means more noise must be added to it. Limiting how much noise is
needed to protect many statistics is an active area of research. The scientific
literature is full of clever tricks to that end. But the best solution is often
the simplest one: calculate fewer statistics.</p>
<p><center>
<img alt="Visual aid showing many round circles representing aggregations, and an arrow
pointing to the right with fewer
circles." src="https://desfontain.es/privacy/images/minimize-the-number-of-aggregations.svg">
</center></p>
<p>OK, this is a bit abstract. How can you decrease the number of statistics you
calculate, in practice? Here are some common strategies.</p>
<ul>
<li>Remove metrics. For example, if you're calculating both the number of page
views and the number of unique visitors… Could you, instead, use only one of
the two?</li>
<li>Remove dimensions. Do you need to calculate the number of visitors per country
<em>and</em> per language? Or would only one of the two get you the information you
need?</li>
<li>Remove time periods. Do you need to calculate the number of unique
visitors in the past day, week, month and year? Or would one or two of these
statistics be enough?</li>
<li>Remove "sliding windows". What's a sliding window? Suppose that every day, you
calculate e.g. the number of visits in the past week. In that case, each data
point will count towards seven separate statistics… Would calculating that
metric only once a week do the trick, instead?</li>
</ul>
<p><center>
<img alt="Visual aid showing fourteen successive days, with many arrows on the top
covering all possible one-week periods within these fourteen days. An arrow
points to the right, where there are only two arrows for two successive
weeks." src="https://desfontain.es/privacy/images/remove-sliding-windows.svg">
</center></p>
<h1 id="split-the-privacy-budget-unevenly">Split the privacy budget unevenly</h1>
<p>This trick is related to the previous one. Suppose that you reduced the number
of aggregations, but you still have several. The idea is to split your total
privacy budget <em>unevenly</em> between them. Say your total privacy budget is
<span class="math">\(\varepsilon=1\)</span> and you have five statistics. You don't have to allocate
<span class="math">\(\varepsilon=0.2\)</span> to each of them! You could instead use <span class="math">\(\varepsilon=0.8\)</span> for
one statistic, and <span class="math">\(\varepsilon=0.05\)</span> for all others. </p>
<p>Splitting the privacy budget unevenly is useful in two common situations.</p>
<ol>
<li>You care much more about the accuracy of some statistics than others. In that
case, you might want to allocate a bigger portion of the privacy budget to
the most important ones.</li>
<li>Some of your statistics are more fine-grained than others. For example,
suppose that you calculate both daily and weekly statistics. On average,
weekly statistics will have 7 times more data than daily statistics: you
could use a budget that is 7 times smaller for them. In doing so, the
relative impact of the noise will be about the same for both.</li>
</ol>
<p><center>
<img alt="Visual aid showing one ε represented as a rectangle split in four equal parts
labeled ε₁, ε₂, ε₃ and ε₄ on the left, and an arrow pointing right to the same
rectangle split in four parts, with the same labels but very different
sizes." src="https://desfontain.es/privacy/images/split-the-privacy-budget-unevenly.svg">
</center></p>
<h1 id="set-aggressive-contribution-bounds">Set aggressive contribution bounds</h1>
<p>Most DP algorithms bound each individual's contribution to each statistic.
For example, if you're counting the number of visits per web page, you need to
bound:</p>
<ul>
<li>the number of different pages each individual can contribute to;</li>
<li>and the number of times each individual can visit each page.</li>
</ul>
<p>How should you pick these bounds? It’s tempting to use generous bounds that will
cover any conceivable input. But this is usually bad for utility: the magnitude
of the added noise grows with the bounds. There is a tradeoff between two
sources of error:</p>
<ul>
<li>larger bounds will lose less data, but require more noise;</li>
<li>smaller bounds might lose more data, but require less noise.</li>
</ul>
<p>Often, reducing the level of noise on your whole dataset is worth truncating a
few outliers. The best cut-off depends on the distribution of your dataset: the
95th percentile might work well for one dataset, while another might do better
with the 98th. In most use cases though, you’ll reach optimal utility when part
of your dataset exceeds the bounds.</p>
<p>Note that some systems don't make you specify these bounds. Instead, they can be
automatically calculated by an auxiliary DP algorithm<sup id="fnref:approxbounds"><a class="footnote-ref" href="#fn:approxbounds">2</a></sup>. But this
operation uses some part of the privacy budget! In that case, if you specify the
bounds by hand instead, you can save that part of the budget, and get less noisy
results.</p>
<p><center>
<img alt="Visual aid showing two histograms of contribution size per user, with an arrow
pointing from the left to the right histogram. A horizontal line representing
the contribution bound splits the histogram in two parts, the top part (which
represents the part of the contribution that is dropped) is of a paler color.
The bound is high on the left and lower on the
right." src="https://desfontain.es/privacy/images/lower-contribution-bounds.svg">
</center></p>
<h1 id="use-public-partitions">Use public partitions</h1>
<p>Most DP pipelines produce many statistics, grouped by <em>partitions</em>. Partitions
are like the buckets in a histogram: in the example where we count the number of
visits per web page, each page is a partition. By default, pipelines only
release statistics for <em>some</em> partitions in the dataset. Typically, the
partitions with most people in them are kept, and the ones with few users are
dropped. This process can reveal information about individuals: it must be done
in a DP way<sup id="fnref:ps"><a class="footnote-ref" href="#fn:ps">3</a></sup>. Like before, we need to use part of the privacy budget for
this, and add extra noise to the statistics to compensate.</p>
<p>You can skip this step by listing all the partitions you want to appear in your
output <em>before</em> you run the query. If you do so using non-private data, it is no
longer necessary to choose partitions in a DP manner. This allows you to save
budget, and return more partitions.</p>
<p>There is a downside: all partitions that you specified will appear in the
output, even if they have little or no data in the dataset. In that case, they
can be extremely noisy. Still, if you can list the partitions you want in
advance, this is often an excellent technique. You can see how it works in
practice in e.g. the <a href="https://codelabs.developers.google.com/codelabs/privacy-on-beam#4">Privacy on Beam codelab</a>. Note that you're
not allowed to look at the private data to build your list of public partitions!</p>
<p><center>
<img alt="Visual aid showing two histograms of number of people per partitions, with an
arrow pointing from the left to the right histogram. A horizontal line labeled
"threshold" splits the left histogram in two: the partitions below the threshold
are paler, to represent them being removed from the output. The right histogram
has no threshold, however additional partitions are added to the output, to
represent the public partitions that are not present in the data but are still
present in the output." src="https://desfontain.es/privacy/images/use-public-partitions.svg">
</center></p>
<h1 id="more-clever-ideas">More clever ideas</h1>
<p>There are a myriad other techniques out there to squeeze more utility out of DP pipelines. Most of them are more complex than the ones listed in this post. Some might be the topic of future blog posts! In the meantime, here are three other, more generic suggestions.</p>
<ul>
<li>You can try looking up your problem in your search engine of choice, and add
"differential privacy". This will often dig up relevant literature. Unless
you're lucky, you won't find a readily available implementation. But you might
get valuable ideas or insights!</li>
<li>You can send a friendly message to one of the communities out there working on
DP tooling. Between <a href="https://groups.google.com/g/dp-open-source-users">Google</a>, <a href="https://www.openmined.org/">OpenMined</a>, or <a href="https://opendp.org/">OpenDP</a>, someone might be happy
to help! (And try to convince you to use their tooling :D)</li>
<li>You can also try to think about how sensitive your problem is to individual
changes in the data. If a single person changes their data, will you come to
different decisions or results? If yes, your use case might be fundamentally
incompatible with DP, and no clever trick will fix that.</li>
</ul>
<hr>
<p>Want to keep learning more about differential privacy? Head over to the <a href="friendly-intro-to-differential-privacy.html">table
of contents</a> of this blog post series to see its other articles.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:osstools">
<p>You did not roll your own tooling, right? Doing so is generally
unwise; implementing DP correctly is much more tricky that you'd expect. There
are some excellent open-source tools out there that you should use instead.
Like <a href="https://github.com/google/differential-privacy">these libraries</a> that my
team at Google published! <a class="footnote-backref" href="#fnref:osstools" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:approxbounds">
<p>Section 5.1.1 of <a href="https://arxiv.org/pdf/1909.01917.pdf">our paper</a> describes such an
algorithm, implemented e.g. <a href="https://github.com/google/differential-privacy/blob/main/cc/algorithms/approx-bounds.h">here</a>. <a class="footnote-backref" href="#fnref:approxbounds" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:ps">
<p>This operation is described in a <a href="almost-differential-privacy.html">previous blog post</a>, and is
also the topic of a <a href="https://arxiv.org/abs/2006.03684">paper I co-authored</a>. <a class="footnote-backref" href="#fnref:ps" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Demystifying the US Census Bureau's reconstruction attack2021-05-26T00:00:00+02:002021-07-26T00:00:00+02:00Damien Desfontainestag:desfontain.es,2021-05-26:/privacy/us-census-reconstruction-attack.html<p>The US Census is moving to differential privacy, after running a successful privacy attack on their 2010 release. Let's look at this attack in more detail!</p><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p></small></p>
<hr>
<p><span class='lettrine'>E</span><strong>very</strong> 10 years, the <a href="https://en.wikipedia.org/wiki/United_States_Census_Bureau">US Census Bureau</a>
asks every American household a series of questions. How many people are living
here? How old are they? What is their race and ethnicity? What is their
relationship to each other?</p>
<p>The collected answers have very high quality, for two reasons. First, <em>all</em>
households have to respond truthfully to these questions, by law. Second, the US
Census Bureau has a legal duty to keep the answers secret for 72 years after
each survey. Both aspects are key to convince everyone to answer truthfully.
Appealing to people's sense of civic duty also helps!</p>
<p>What is the data used for, then? The Census Bureau aggregates it and publishes
statistics about the US population. A <em>lot</em> of statistics: in 2010, it released
over 150 <em>billion</em> statistics about the US population. These statistics then
have many applications: scientific research, informing public policy,
determining election districts, etc.</p>
<p>Confidentiality is central to the Census Bureau's mission.
<a href="https://www.law.cornell.edu/uscode/text/13/9">By law</a>, they have to make sure
that nobody can use their data to find out information about individuals.
Disclosing such information even comes with
<a href="https://www.law.cornell.edu/uscode/text/13/214">criminal penalties</a>! This has
led to the creation of an entire field of study: <em>statistical disclosure
control</em>. It predates even the oldest definition of privacy mentioned on this
blog (<a href="k-anonymity.html"><em>k</em>-anonymity</a>).</p>
<p>How did statistical disclosure control work in practice? Before 1990, the method
was pretty blunt: they removed the tables with fewer than five individuals or
households in them. Then, from 1990 to 2010, the Census Bureau used a more
complex technique called <em>swapping</em>. Swapping works in two steps. First, it
selects households at random in small geographic areas: those are the ones most
at risk of re-identification. Then, it exchanges records between these
households and others before generating the statistics. The details of this
swapping process were secret, to make it harder for people to design attacks.</p>
<p>Nowadays, the Census Bureau is moving towards formal notions: the statistics
published for the 2020 Census will be differentially private. If you've read the
<a href="differential-privacy-awesomeness.html">previous articles</a> of this blog, this might not surprise you.
Differential privacy (DP) is designed to solve this exact problem: publishing
statistics without revealing information about individuals. But this choice was
far from obvious! Adding noise to statistics is quite scary for data
users<sup id="fnref:users"><a class="footnote-ref" href="#fn:users">1</a></sup>. Using DP for such a complex release is also unprecedented.</p>
<p>So, what convinced the Census Bureau to take this decision? Their scientists ran
an <em>attack</em> on some of the 2010 data, to better understand the privacy risks.
And they realized that the attack was much more successful than they expected.
The attack is simple, not very expensive, and pretty scary. The scientists then
measured how much more swapping it would take for the attack to fail… and
observed that the utility loss would be catastrophic. Older techniques like
suppression were also ineffective. The only way to mitigate the risks <em>and</em>
limit accuracy loss was differential privacy.</p>
<p>This choice, of course, has other benefits. Differential privacy provides
<a href="differential-privacy-awesomeness.html#quantify">quantifiable guarantees</a>. It <a href="differential-privacy-awesomeness.html#composition">composes well</a>, and
protects even against <a href="why-not-differential-privacy.html">very powerful attackers</a>. It's also good for
transparency: for the first time, the Census Bureau will be able to publish the
details of their process. But these nice theoretical properties weren't the main
factor in their decision. Instead, the choice of DP was <em>pragmatic</em>: it worked
best to prevent realistic attacks without sacrificing too much utility.</p>
<p>In this blog post, we'll take a look at the details of this compelling attack,
at the core of this decision. The attack has two stages: <em>reconstruction</em> and
<em>reidentification</em>.</p>
<h1 id="reconstruction">Reconstruction</h1>
<p>The first step in the attack is to reconstruct records, using statistical data.
The statistical data is as follows. In each geographic area (like <a href="https://en.wikipedia.org/wiki/Census_block">Census
blocks</a>), we can get the answers to
questions like: how many people of age 47 live there? How many people between
the ages of 25 and 29 self-identify as white? How many households with 2 adults
and 2 children are there?</p>
<p>How can we use this kind of information and reconstruct the original records?
Let's take a simplified example. Say that a hypothetical Census block has four
people in it.</p>
<table>
<thead>
<tr>
<th style="text-align: center;">Age</th>
<th style="text-align: center;">Race<sup id="fnref:race"><a class="footnote-ref" href="#fn:race">2</a></sup></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">17</td>
<td style="text-align: center;">White</td>
</tr>
<tr>
<td style="text-align: center;">17</td>
<td style="text-align: center;">Asian</td>
</tr>
<tr>
<td style="text-align: center;">43</td>
<td style="text-align: center;">White</td>
</tr>
<tr>
<td style="text-align: center;">47</td>
<td style="text-align: center;">Asian</td>
</tr>
</tbody>
</table>
<p>Now, suppose that we have the following statistical data about this Census
block.</p>
<ol>
<li>There are four people in total.</li>
<li>Two of these people have age 17.</li>
<li>Two of these people self-identify as White.</li>
<li>Two of these people self-identify as Asian.</li>
<li>The average age of people who self-identify as White is 30.</li>
<li>The average age of people who self-identify as Asian is 32.</li>
</ol>
<p>This data is statistical in nature: these numbers are all <em>aggregated</em> over
several people (here, two). Yet, it's not difficult to guess what the original
table looks like based on the numbers.</p>
<p>Not obvious yet? Take the two people aged 17. Points 1, 3 and 4 tell us that:</p>
<ul>
<li>either they both self-identify as White,</li>
<li>either they both self-identify as Asian,</li>
<li>either one of them self-identifies as White and the other as Asian.</li>
</ul>
<p>The first option is impossible: if they both self-identified as White, then
their average age should be 17, not 30 (point 5). The second option is also
impossible, for the same reason (with point 6). So the third option is correct.
We now know the first two records, and we can find the age of both others using
the average age of each race group. It's like a fun puzzle!</p>
<p>That's the idea behind reconstruction attacks: taking statistical information
and using simple reasoning to reverse-engineer the original records. Of course,
when we have billions of statistics to work with, we don't do this by hand.
Instead, we convert the data into a massive <a href="https://en.wikipedia.org/wiki/System_of_linear_equations">system of equations</a>.
Each piece of individual information is an unknown variable. The data gives us
the relationships between them. Then, we can use a big computer to find a
possible solution to this large system. This solution, in turn, gives us
reconstructed records.</p>
<p>So, the team at the Census Bureau did exactly that, with statistical data from
the 2010 Census. They transformed it into many equations, and used <a href="https://www.gurobi.com/">Gurobi</a> to
reconstruct the raw data. The records they obtained matched <strong>46%</strong> of the
original records exactly. That's pretty impressive! Especially since only a
small fraction of the statistics were used in the attack (6.2 billion out of 150
billion). Swapping was not enough to prevent accurate reconstruction.</p>
<p>At first glance, that result looks pretty scary. But let's take a step back: how
bad is it really? What does this 46% number actually tell us?</p>
<h1 id="how-bad-is-reconstruction">How bad is reconstruction?</h1>
<p>Imagine that a given Census block has a particularly homogeneous population: out
of 100 habitants, 95 all self-identify as White, and are evenly distributed
between the ages of 20 and 39<sup id="fnref:homogeneous"><a class="footnote-ref" href="#fn:homogeneous">3</a></sup>. The other 5% of people in this
block do not belong to this demographic, and we don't have any information about
them.</p>
<p><center>
<img alt="Visual representation of a hypothetical homogeneous population. A large block
is labeled "Majority population, Self-Identifies as White, Ages evenly
distributed between 20 and 39"; a smaller block underneath is labeled "Minority
population, no demographic information"." src="https://desfontain.es/privacy/images/homogeneous-population.svg">
</center></p>
<p>Can we "reconstruct" this dataset with high accuracy? Yes, and it will be easier
than we expect: simply use the data on the majority group, and ignore the
minority population. In practice, for each age between 20 and 39, we output 5
records with this age, and White as a race.</p>
<p>Given the statistics on this block, it's likely that our naive guess is pretty
close to the truth. After all, only 5% of people don't belong to the majority
demographic, and we know the distribution of the other 95%. Recall the success
metric from the previous section: the percentage of matching records between
real data and reconstructed data. According to this metric, our naive strategy
performs very well! Accuracy is about 95%, if the age distribution is very
uniform.</p>
<p>Has this process revealed sensitive information? Well… not really, right? All
reconstructed records are identical across many people. So they don't seem to
reveal very sensitive information… And the people in demographic minorities are
safe from the attack.</p>
<p>It's also not clear yet how problematic these reconstructed records are. All we
get is a list of records that are compatible with the published statistics. But
how do we know which ones are actually correct? And how can we use them to learn
something new and sensitive about individuals?</p>
<p>These questions show that on its own, the 46% number from the Census Bureau
doesn't tell us much. But the Census Bureau didn't stop at reconstruction. The
attack has a second step, <em>re-identification</em>, and this one gets much scarier
results. Let's take a look at how it works.</p>
<h1 id="re-identification">Re-identification</h1>
<p>In the Census Bureau attack, the reconstruction step outputs records with five
characteristics: Census block, age, sex, race, and ethnicity. The idea of the
re-identification attack is simple. First, they buy the kind of <em>commercial
data</em> that an attacker could have access to. Second, they <em>link</em> this
information with reconstructed records.</p>
<p>This "commercial data" is exactly what it sounds like: information about people,
compiled by random companies, and made available to anyone who pays for it (or
<a href="https://en.wikipedia.org/wiki/2017_Equifax_data_breach">steals it</a>). These
companies, called "data brokers", are largely unregulated in the US. Their
privacy practices are about as awful as you can imagine<sup id="fnref:brokers"><a class="footnote-ref" href="#fn:brokers">4</a></sup>. For their
attack, Census obtained data from five different companies. Their goal was to
simulate what an attacker would likely have access to in 2010.</p>
<p>These commercial datasets typically contain people's names, associated with
demographic information: location of residence, age (or date of birth), and
sex<sup id="fnref:sex"><a class="footnote-ref" href="#fn:sex">5</a></sup>. We will use these three characteristics to <em>re-identify</em>
reconstructed records.</p>
<p>The technique is simple. We look at each record of the commercial dataset, one
after the other. And we try to match this record's characteristics with the
reconstructed record. Is there a single reconstructed record with the same
location, age, and sex? If so, we link both records together. Here is a
visualization of that process (with made-up data). The commercial
dataset is in blue, the reconstructed Census records are in green.</p>
<p><center>
<img alt="Visual representation of a join between a record from a commercial dataset,
and a reconstructed Census record. The commercial record has a name, James Link
(labeled Identity), a postal address, a date of birth, and a sex. The
reconstructed Census record has five fields: Census block ID, age, sex, race,
and ethnicity. Location, age and sex form a "join key", and allow both records
to be linked. The race and ethnicity are labeled "newly learned
information"." src="https://desfontain.es/privacy/images/re-identification.svg">
</center></p>
<p>Linking datasets in this way achieves two goals.</p>
<ol>
<li>It confirms that the reconstructed record corresponds to a specific
individual. In fact, it allows us to <em>re-identify</em> the reconstructed record.
Here, the commercial data tells us that this person is James Link.</li>
<li>It gives us <em>more</em> information about this person, which the commercial data
didn't have. Here, we learn James Link's race and ethnicity.</li>
</ol>
<p>How do we quantify the success of this attack? We can look at two numbers.
First, how many records can be linked between datasets in this way? Second, out
of these linked records, how many are <em>accurate</em>? After all, some of these
re-identifications might be wrong: both datasets might be incomplete or
inaccurate. The percentage of correct answers is the <em>precision</em> of the attack:</p>
<div class="math">$$
\text{precision} = \frac{\text{true re-identifications}}{\text{number of linked records}}.
$$</div>
<p>So how does the attack perform? Census Bureau scientists linked 138 million
records between both datasets. And their average precision was 38%. This means
that the attack <strong>successfully re-identified 52 million records</strong>. Scary!</p>
<p>One could argue that the attacker can't know <em>which</em> records are correctly
re-identified. Some of them will be false positives! After all, 38% aren't great
odds. But the analysis from Census Bureau scientists doesn't stop there. Two
further considerations make the attack even more compelling.</p>
<ol>
<li>The precision of re-identification goes <em>up</em> for people in small Census
blocks: it's <strong>72%</strong> on the smallest Census blocks (in which 8 million people
live). This makes sense: statistics across a few people are more revealing
than aggregates over large groups. It's not surprising, but it's still bad
news: it means that folks in minority populations are more at risk of being
re-identified. But disclosure avoidance is precisely trying to protect these
people!</li>
<li>The precision also goes up if one has better-quality data. Census Bureau
scientists use a neat trick to find worst-case guarantees: they use the raw
Census data itself as an approximation of the best possible data an attacker
could find. Using this, they show that the global precision increases to 75%,
and even goes up to <strong>97%</strong> on small Census blocks.</li>
</ol>
<p>Is this assumption of high-quality data unrealistic? No, for two reasons.</p>
<ul>
<li>The commercial data used in the attack is what someone could have had access
to in 2010 — more than 10 years ago. Data brokers have much better data
available for sale by now.</li>
<li>The re-identification step only requires high-quality data <em>about our
targets</em>. The attack works just fine even if we're trying to re-identify a
single person, or a handful of people. With high-quality information about
them, the precision goes up to these worst-case numbers.</li>
</ul>
<p>There's nothing preventing businesses from running this attack on their
employees or customers. In this kind of context, access to high-quality
data isn't an optimistic assumption: it's a given.</p>
<h1 id="conclusion">Conclusion</h1>
<p>This attack is bad news.</p>
<p>It proves two points beyond doubt. First, anyone can reconstruct Census records.
Many of these reconstructed records are accurate, especially in small Census
blocks. This does not require massive computing power: you can run
<a href="https://www.nytimes.com/2018/12/05/upshot/to-reduce-privacy-risks-the-census-plans-to-report-less-accurate-data.html">smaller-scale attacks</a>
in minutes on your average laptop. Second, high-quality data about some people
is enough to re-identify their Census records. This re-identification has high
precision, and reveals previously-unknown information about these people.</p>
<p>In the attack, the attacker could learn race and ethnicity information. Data
brokers might want to do this to augment their datasets, for example. But the
risks can get even more tangible. Remember: the attack only used a small
fraction of all published statistics. The targeted tables had only demographic
information. What if someone were to attack household tables instead? This could
likely reveal intimate details about the people you live with. Do you have a
have a partner of the same sex? Children of a different race? Anyone with basic
information about you — employer, acquaintance, data broker… — might find out.</p>
<p>Some of that information is particularly sensitive, and otherwise unavailable
via data brokers. For example, reconstruction also works on children's data,
which is illegal to sell in the US. Someone could combine household
reconstruction with re-identification, and locate kids of a specific race, sex
and age range in a given area. No need to spell out the possible harms this
could enable.</p>
<p>Risks to individuals can translate to risks to data quality in the long run.
Suppose Census data can no longer be considered as confidential. How will it
impact people's willingness to answer future Census surveys? Minority groups are
already <a href="../PDFs/CensusBarriersReport.pdf">more likely to have privacy concerns</a> about their answers. This
will get worse if the Census Bureau can no longer guarantee the confidentiality
of the data.</p>
<p>This explains the Census Bureau's decision to move to differential privacy.
Everything we described is just one possible attack, and it used only a fraction
of the released data. The Census must protect all other attacks that people
might come up with in the future! And that's exactly what differential privacy
provides: provable privacy guarantees, even against attacks that haven't
invented yet.</p>
<hr>
<p>If you'd like to learn more about differential privacy, I have good news for
you: this blog also contains an entire <a href="friendly-intro-to-differential-privacy.html">article series</a> about this
notion, introducing it in friendly, non-technical terms. Head over to the <a href="friendly-intro-to-differential-privacy.html">table
of contents</a> to see the other posts!</p>
<hr>
<p><small></p>
<p>Nice, you made it all the way to the end of this article! Here are some more
links and disclaimers.</p>
<p>This article is almost entirely sourced using documents from a <a href="https://www.brennancenter.org/our-work/court-cases/alabama-v-us-dept-commerce">lawsuit</a>. The
main one is this <a href="../PDFs/AlabamaLawsuitDocuments.pdf">declaration from John Abowd</a>, the Census Bureau's
chief scientist. It explains the history and context behind the use of DP for
the 2020 Census. It starts on page 85 of <a href="../PDFs/AlabamaLawsuitDocuments.pdf">this PDF</a>, and Appendix B
(starting on page 147) describes the attack in more detail. Some of the numbers
also come from Abowd's <a href="../PDFs/AbowdSupplementalDeclaration.pdf">supplemental declaration</a>.</p>
<p>This last declaration has been filed as a response to a filing by a couple of
demographers. They make an <a href="../PDFs/RugglesReconstructionExperiment.pdf">argument</a> similar to the one outlined in
the second part of this article: you can reconstruct records by picking them
randomly, so reconstruction doesn't mean anything. Hopefully, this post managed
to convinced you that this argument has two flaws: it ignores the higher risks
for minority groups, and it fails to address re-identification, the second part
of the attack.</p>
<p>Still hungry for more legal filings? Many leading data privacy experts filed an
<a href="../PDFs/CensusDifferentialPrivacyAmicusBrief.pdf">amicus brief</a> supporting the use of DP for the 2020 Census. It's clear,
concise, and makes a lot of great points. Worth a read!</p>
<p>In case this wasn't obvious, this article made a <em>lot</em> of simplifications. I
rounded all numbers so they would look nicer. I optimized the reconstruction
example for clarity and fun, not for accuracy: in particular, the Census Bureau
doesn't actually release statistics like "average age". The real attack doesn't
use only a system of equations for reconstruction, but inequalities as well; I
also suppose that there is some optimization done, but the details aren't
public. The linking attack has some additional logic to do fuzzy matching. And
I'm sure I made other shortcuts along the way. If you're looking for more
accurate information, you will probably find it in the documents linked above.</p>
<p>There's a lot more to say about <a href="https://www.census.gov/content/dam/Census/library/working-papers/2018/adrm/Disclosure%20Avoidance%20Techniques%20for%20the%201970-2010%20Censuses.pdf">swapping</a>, too. For example, it has surprising
negative effects on data quality! Since folks in demographic minorities in each
location are more likely to have their record swapped, the process biases the
data: it makes all areas seem more homogeneous than they actually are. Recall
that the details of swapping are secret: data users couldn't quantify such
effects to take them into account in their analysis! Interestingly, the attack
run by Census Bureau scientists ignored swapping entirely. A more clever attack
might take it into account, and attempt to reverse it. This could make the
attack even more accurate, especially for folks in demographic minorities.</p>
<p>Finally: I have not been involved in any way with Census work. I'm thankful to
Aloni Cohen, Cynthia Dwork, Thomas Steinke, Kunal Talwar, and Yuan Yuan Zheng
for helpful comments and suggestions on drafts of this post. Of course, if there
are inaccuracies or errors left, that's entirely on me — please let me know if
you find any!</p>
<p></small></p>
<div class="footnote">
<hr>
<ol>
<li id="fn:users">
<p>The people <em>using the data</em>: scientists, people drawing electoral
district boundaries, public agencies, businesses, etc. <a class="footnote-backref" href="#fnref:users" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:race">
<p>We're using the
<a href="https://www.census.gov/topics/population/race/about.html">classification from the Census</a>
here. Census data also has a separate
<a href="https://www.census.gov/topics/population/hispanic-origin/about.html">Hispanic origin</a>
field, called "ethnicity". Don't ask me to explain this, I have no idea. <a class="footnote-backref" href="#fnref:race" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:homogeneous">
<p>This hypothetical Census block is basically a tech company. <a class="footnote-backref" href="#fnref:homogeneous" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:brokers">
<p>To learn more about this industry,
<a href="https://www.eff.org/wp/behind-the-one-way-mirror#Data-brokers">this EFF paper</a>
is a pretty solid resource. <a class="footnote-backref" href="#fnref:brokers" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:sex">
<p>Or gender, depending on the dataset. These are
<a href="https://en.wikipedia.org/wiki/Sex_and_gender_distinction">not the same thing</a>,
but the Census asks about sex, so we'll assume the commercial datasets use the
same notion. This inevitably introduces errors and feels a bit icky. <a class="footnote-backref" href="#fnref:sex" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Why not differential privacy?2021-03-30T00:00:00+02:002021-09-27T00:00:00+02:00Damien Desfontainestag:desfontain.es,2021-03-30:/privacy/why-not-differential-privacy.html<p>What does it mean for an algorithm to <em>not</em> be differentially private?</p><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p></small></p>
<hr>
<p><span class='lettrine'>S</span><strong>ome</strong> algorithms are not differentially
private, but still claim to perform anonymization. Such mechanisms are common,
both in the academic literature and in industry. Explanations on why they still
preserve some notion of privacy vary.</p>
<ul>
<li>They might include some ad hoc protections against entire classes of attacks.</li>
<li>They might aggregate the data to a point where the statistics "obviously" seem
safe.</li>
<li>They might use some other metric for data leakage, like entropy or mutual
information.</li>
</ul>
<p>How to get an idea of how robust these proposals are? In this post, I'll suggest
a somewhat provocative approach: we'll try to analyze them <em>in the language of
differential privacy</em>. We're going to ask the following question: <em>why</em> isn't
a given mechanism differentially private?</p>
<p>I'll need a straw man to get the discussion going. Meet Paille.</p>
<p><center>
<img alt="A drawing of a friendly straw man" src="https://desfontain.es/privacy/images/paille.png">
</center></p>
<p>Paille (it's pronounced <span class="math">\(\pi\)</span>) has an algorithm <span class="math">\(A\)</span>. They believe that <span class="math">\(A\)</span>
performs anonymization: it protects the data of individuals in its input. Their
line of argument starts with: </p>
<blockquote>
<p>It's not differentially private, <em>but</em>… [insert long explanation here]</p>
</blockquote>
<p>Rather than focusing on the explanation itself, let's dig into <em>why</em> that
algorithm is not DP. First, what does it mean for an algorithm to not be DP?
Let's take <a href="differential-privacy-in-more-detail.html">the definition of differential privacy</a> and negate it. If
<span class="math">\(A\)</span> isn't DP, then there are databases <span class="math">\(D_1\)</span> and <span class="math">\(D_2\)</span> differing in only one
individual, such that the ratio:
</p>
<div class="math">$$\frac{\mathbb{P}\left[A(D_1)=O\right]}{\mathbb{P}\left[A(D_2)=O\right]}$$</div>
<p>
gets arbitrarily large for varying possible outputs <span class="math">\(O\)</span>. Remember: we called
this ratio the <a href="privacy-loss-random-variable.html">privacy loss</a> before.</p>
<p>Suppose that an attacker is hesitating between <span class="math">\(D_1\)</span> and <span class="math">\(D_2\)</span>: they know all
the database, except the data of one single individual. Then it's possible that
by looking at the output of <span class="math">\(A\)</span>, the attacker knows <em>for sure</em> what the data of
this person is.</p>
<p>That sounds… not great. In fact, that sounds exactly like what we were trying to
avoid. Why is this OK? Oh, wait, I have an idea. What if this only happens <em>very
rarely</em>?</p>
<h1 id="averaging-the-privacy-loss-across-outputs">Averaging the privacy loss across outputs</h1>
<p>Differential privacy is a <em>worst-case</em> property: it must hold for <em>every</em>
possible output <span class="math">\(O\)</span>. So if there's the slightest chance that the ratio of
probabilities is unbounded, we don't get DP. Yet, we might be able to say
« unless we're extraordinarily unlucky, the DP property holds ». In fact, we've
<a href="almost-differential-privacy.html">done this before</a>, when we introduced <span class="math">\((\varepsilon,\delta)\)</span>-DP. That
could be good enough!</p>
<p>We saw that <span class="math">\((\varepsilon,\delta)\)</span>-DP allows a small possibility of catastrophic
failure: the privacy loss can sometimes be <em>infinite</em>. To avoid this, we can
<em>average</em> the privacy loss across all possible outputs instead. Some variants of
DP even let us choose what <em>kind</em> of averaging function we want to use<sup id="fnref:renyi"><a class="footnote-ref" href="#fn:renyi">1</a></sup>.</p>
<p>So, Paille, is this what's happening here? Do we have differential privacy for
almost all possible outputs? Or is the average privacy loss bounded by some
reasonable value?</p>
<blockquote>
<p>Eh… not exactly. The privacy loss can be really large even if we average it
across all possible outputs.</p>
</blockquote>
<p>Oh, OK. Well, let's see what else could happen. What if, instead of averaging
the privacy loss across outputs, we average it across people?</p>
<h1 id="averaging-the-privacy-loss-across-people">Averaging the privacy loss across people</h1>
<p>Differential privacy gives the <em>same</em> protection to all individuals. The
guarantees on the privacy loss apply to everyone. Is that always necessary? In
some cases, it might be reasonable to say that some people need more privacy
protection than others. For example, folks from at-risk populations might need a
smaller <span class="math">\(\varepsilon\)</span> than majority groups.</p>
<p>Another possible approach is to protect <em>almost all</em> individuals, without
specifying <em>which ones</em>. To do so, we first need to model the population
according to a probability distribution. Then, we say « with high probability,
someone sampled from this distribution is protected ». Unlucky people might not
get any protection, but these are hopefully very rare<sup id="fnref:random"><a class="footnote-ref" href="#fn:random">2</a></sup>.</p>
<p>This is a bit like <span class="math">\((\varepsilon,\delta)\)</span>-DP: there is a small chance that
things go wrong. We could, instead, average the privacy loss <em>across people</em>.
Like before, it would avoid the possibility of infinite risk for some
individuals. This is much less robust than the previous class of definitions,
though. First, some people might <em>never</em> get good protection, if their data is
unusual. Second, it requires us to model our population with a probability
distribution. This is hard to do! And if our model is wrong, more folks might
be at risk than we expected.</p>
<p>Still, though, it's something. Paille, does your algorithm <span class="math">\(A\)</span> behave this way?</p>
<blockquote>
<p>Hmmm… no. It seems that the privacy loss is very large for more than a few
individuals. So averaging it doesn't bring much.</p>
</blockquote>
<p>Arg. Well… If you're not protecting individuals, maybe you're protecting some
other property?</p>
<h1 id="changing-the-protected-property">Changing the protected property</h1>
<p><a name="privacy-units"></a></p>
<p>With DP, the attacker tries to distinguish between databases differing in <em>one
person</em>. This means that we protect everything about any single individual.
Sometimes, though, getting to this level of protection seems like an impossible
task.</p>
<p>For example, suppose the input database is growing over time: every day, we get
new data from the users of an app. We want to publish an anonymized version of
this daily data every day. Each daily release might be differentially private…
But the total privacy loss of a given user over time is unbounded: the same
person might use the app every day for a long time.</p>
<p>This is better than nothing, though: we can still claim that we're protecting
all contributions of each user in every single day. Capturing this idea is easy:
we can redefine "neighboring datasets" to differ in the data of a single person
in a single day.</p>
<p>We can also extend this idea to other properties that we want to protect. Maybe
finding out that someone is in our database might not be that sensitive. But
finding out the value of a specific field might be problematic! In this case, we
can adapt the definition of DP, and have the two databases differ only in this
field for a single individual.</p>
<p>Paille, can you capture the properties of your algorithm <span class="math">\(A\)</span> this way? If it's
too hard to get formal privacy guarantees for individuals, can you do it for
smaller "units of privacy"?</p>
<blockquote>
<p>Erm… it doesn't look like it. Even when the "unit of privacy" is smaller, the
privacy loss is still too high to be meaningful.</p>
</blockquote>
<p>Well, this doesn't look great. But let's persevere and try one last thing.
What if we assume the attacker is uncertain about the initial dataset?</p>
<h1 id="assuming-a-weaker-attacker">Assuming a weaker attacker</h1>
<p>When using DP, we compare the output of the algorithm on two databases that
differ in a single user. Implicitly, we assume that the attacker knows the data
of <em>everyone else</em> in the database. What if we relax this assumption?</p>
<p>Doing this seems reasonable. After all, the only realistic way an attacker could
know about everyone in a database is by having direct access to the database…
And then there's not much left to protect. Some variants of DP attempt to
formalize this idea. To do this, they capture the attacker's uncertainty using a
<em>probability distribution</em>. The neighboring databases are no longer fixed:
they're sampled from this distribution, conditioned on the data of a specific
user.</p>
<p>The variants of differential privacy obtained this way<sup id="fnref:noiseless"><a class="footnote-ref" href="#fn:noiseless">3</a></sup> have two
major problems.</p>
<ul>
<li>First, they don't <a href="differential-privacy-awesomeness.html#composition">compose</a>. Say two algorithms are "private" if
an attacker has limited background knowledge. Each output, in isolation,
doesn't leak too much information. Both outputs <em>combined</em>, though, might not
be private at all, even under the same assumption.</li>
<li>Second, these variants need us to model the database as a probability
distribution. This distribution is supposed to capture the attacker's
uncertainty… So you have to put yourself in the shoes of each possible
attacker and model their knowledge somehow. This is difficult and very
brittle: if you misjudge their knowledge even slightly, all privacy properties
might break down.</li>
</ul>
<p>Because of this<sup id="fnref:reasons"><a class="footnote-ref" href="#fn:reasons">4</a></sup>, assuming a weaker attacker can be kind of a dangerous
road. Paille, does your algorithm <span class="math">\(A\)</span> satisfies one of these variants? It
wouldn't be enough to fully convince me: I'd also need take a long look at the
underlying assumptions, and at how you're using it in practice. Nonetheless,
it'd be a start, and it'd be better than nothing, I guess.</p>
<blockquote>
<p>Well, let me check. Modeling the attacker's uncertainty is difficult, but…
doing that doesn't give me convincing results either. I can make unrealistic
assumptions on my data, and then it sort of works. But if I try to model the
attacker in a more realistic way, I don't get great numbers at all.</p>
</blockquote>
<p>That's… unfortunate.</p>
<h1 id="whats-left">What's left?</h1>
<p>Let's recap what we know so far about Paille's algorithm <span class="math">\(A\)</span>. If we negate all the
relaxations we've seen so far, what do we have left?</p>
<p>An attacker who looks at the output of <span class="math">\(A\)</span>:</p>
<ul>
<li>can retrieve very <em>fine-grained</em> information</li>
<li>about <em>many individuals</em></li>
<li>even if the attacker is not particularly <em>lucky</em></li>
<li>and only has <em>limited knowledge</em> about the data.</li>
</ul>
<p>This is <strong>not good</strong>! But this is the direct conclusion of the discussion so
far. Paille's mechanism not being DP didn't seem so bad at first, after all, DP
is quite a high bar. But if we can't say <em>anything</em> about <span class="math">\(A\)</span> in the language of
DP, even if we relax the definition a lot, then this is pretty damning. No need
to dive deep into the original rationale for why <span class="math">\(A\)</span> might be safe: we just
showed it isn't.</p>
<p>Or, rather, we are unable to show that it is. This will be the last resort of
people defending their custom anonymization method: « I can't prove that it's
safe, but I still argue that it is. Prove me wrong! Show me an attack that
works. » Reversing the burden of proof this way is, of course, a red flag. If
you're anonymizing my data, you should have to convince me that what you're
doing is safe, not the other way around.</p>
<p>Further, experience shows that if someone does find an attack, that won't be
enough to end the debate. In practice, people slap a patch or two on their
algorithm, and go right back to proclaiming its safety. The history of computer
security is littered with such examples: people patch systems after an attack is
discovered, but shortly after, a minor change to the attack proves successful.
The early days of data privacy were no different. I hope that we learn from this
past, and focus future efforts on stronger notions with provable guarantees!</p>
<p>So, next time you encounter a non-DP algorithm… Why don't you insist that its
authors explain to you <em>why</em> it isn't DP?</p>
<h1 id="final-note">Final note</h1>
<p>There are many more variants and extensions of DP beyond those mentioned in this
post. In fact, a colleague and I wrote a whole <a href="https://arxiv.org/abs/1906.01337">survey paper</a> about it!
In this paper, we classify all these variants, list their properties, and
provide intuitions for each. For a short overview of this work, you can check
out the <a href="https://www.youtube.com/watch?v=P2GyJYb9FOc">recording of the talk</a> I gave about it at <a href="https://petsymposium.org/">PETS</a> last
summer.</p>
<hr>
<p>The next article in this branch is about the US Census Bureau's <a href="us-census-reconstruction-attack.html">reconstruction
attack</a>. Or you can also go to the <a href="friendly-intro-to-differential-privacy.html">table of contents</a>
of this blog post series to pick something else to read!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:renyi">
<p>This is <a href="https://arxiv.org/abs/1702.07476">Rényi DP</a>, a definition
often used for machine learning applications. Its additional parameter
<span class="math">\(\alpha\)</span> determines which averaging function is used: <span class="math">\(\alpha=1\)</span> bounds the
<a href="https://en.wikipedia.org/wiki/Geometric_mean">geometric mean</a> of the ratio,
<span class="math">\(\alpha=2\)</span> bounds the <a href="https://en.wikipedia.org/wiki/Arithmetic_mean">arithmetic mean</a>,
<span class="math">\(\alpha=3\)</span> bounds the <a href="https://en.wikipedia.org/wiki/Root_mean_square">quadratic mean</a>,
etc. <a class="footnote-backref" href="#fnref:renyi" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:random">
<p>This is <a href="https://arxiv.org/abs/1112.2680">random DP</a>, though a couple
of other variants do essentially the same thing. <a class="footnote-backref" href="#fnref:random" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:noiseless">
<p>Like <a href="https://eprint.iacr.org/2011/487">noiseless privacy</a>, named
this way because even algorithms that don't add any noise to data can be
considered private under this variant. <a class="footnote-backref" href="#fnref:noiseless" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:reasons">
<p>… and because of other complicated technical reasons I won't get
into here, though I co-authored
<a href="https://arxiv.org/abs/1905.00650">an entire paper</a> about these problems. <a class="footnote-backref" href="#fnref:reasons" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Converting my PhD thesis into HTML2021-03-04T00:00:00+01:002023-12-28T00:00:00+01:00Damien Desfontainestag:desfontain.es,2021-03-04:/privacy/latex-to-html.html<p>A story of false hopes, perseverance, pain, and futility.</p><p>Finishing a PhD is a weird emotional experience. All the hard work, the joys,
the pains, the pulled hairs, everything gets condensed into a <a href="/thesis.pdf">scary-looking
PDF</a> and then you're just… done? What? This makes no sense
whatsoever. Or rather, this makes sense on paper, but then you feel this weird
sense of grief somehow. And you're not quite at the acceptance stage yet. So
instead, you decide to deal with those feelings in a perfectly normal and
healthy way, and you embark on a journey to compile said thesis <a href="/thesis/">into a series
of HTML pages</a>.</p>
<p>HTML, by the way, is a much better way of disseminating information than PDF.
Pretty much all of recent scientific research is recorded in PDF files, for
historical reasons that are largely irrelevant today. PDFs are difficult to
browse, impossible to read on a phone, uncomfortable to read on a tablet,
hostile to screen readers, impractical to search engines, and the list goes on.
It's just a terrible format, unless you're trying to print things on paper.
Printing things is a perfectly reasonable thing to do, but that's really not the
main use case we should be optimizing for.</p>
<p>Anyway. I <a href="/thesis/">converted my thesis to HTML</a> and this is my story. A story
of false hopes, perseverance, pain, and futility. I hope this can be useful to
other people, as a guide on how to do this for your own thesis or large &
complex LaTeX documents, or as an encouragement to do something better with your
time instead.</p>
<h1 id="false-hopes">False hopes</h1>
<p>"Convert LaTeX to HTML", I type in my search engine of choice. Ooooh, I have
options! There's <a href="https://pandoc.org/">pandoc</a>, <a href="https://ctan.org/pkg/lwarp">lwarp</a>, <a href="https://dlmf.nist.gov/LaTeXML/">LaTeXML</a>, <a href="https://tug.org/tex4ht/">TeX4ht</a>, and probably others.
This looks excellent. Converting LaTeX to HTML is clearly a problem that other
people have already solved for me before. I will just have to run an existing
tool, and iron out the kinks.</p>
<p>I download the tools in question, run them on my thesis, and look at the initial
results. <a href="https://tug.org/tex4ht/">TeX4ht</a> fires off a bunch of compilation
errors and warnings, but it outputs something that kinda looks reasonable from a
distance. All others fail completely. So I go, "OK, let's try to fix the TeX4ht
problems, to get a feeling for how difficult this is". It turns out not to be
too difficult to fix most common issues, <a href="https://tex.stackexchange.com/questions/tagged/tex4ht">LaTeX Stack
Exchange</a> answers most of
my questions, so I make progress. I also notice that there is a nice-looking
build system for TeX4ht called <a href="https://github.com/michal-h21/make4ht">make4ht</a>,
which looks really nifty, I imagine it's going to be similar to
<a href="https://mg.readthedocs.io/latexmk.html">latexmk</a>, which I love.</p>
<p>So, things are going alright. I make progress. Here are some of the problems I
found at first and how I fixed them.</p>
<ul>
<li>A bunch of packages or commands don't make much sense in an HTML context: page
breaks, PDF anchors, page numbers, floats, landscape layouts, margins or other
types of spacing… Some of them (like <code>floatrow</code>) throw compilation errors,
most are simply ignored. I made a pass at all the packages I used and removed
the ones that were obviously irrelevant for HTML.</li>
<li>One special case is <code>longtable</code>: since a regular table can be as long as you
need it to be in HTML, you also don't need it. Replacing it by a regular
<code>tabular</code>, and <code>ThreePartTable</code> (from <code>threeparttablex</code>) by the regular
<code>threeparttable</code> fixed the problem.</li>
<li>Importing an image originally stored in a PDF rendered it into a tiny
unreadable vignette. Adding a "config file" with some <a href="https://tex.stackexchange.com/a/46210">dark magic in
it</a> did the trick. </li>
<li>SVGs generated with <code>tikzpicture</code> by were very wrong (missing text, blank
graphs…). Apparently the "driver" included in htlatex is not good, but for
some reason it's <a href="https://tex.stackexchange.com/a/488741">still in use</a>.
Including the line that calls a different driver wasn't enough, even though
the file was already present in my system, I still got some bugs (text not at
the right place). Importing the file <a href="https://github.com/michal-h21/dvisvgm4ht/blob/master/pgfsys-dvisvgm4ht.def">directly from
GitHub</a>
worked.</li>
<li>Some commands don't work for reasons I didn't really understand, but are
easily fixable: for example, <code>\notin</code> works fine with <code>pdflatex</code>, but <code>tex4ht</code>
complains about it. Replacing it by <code>\not{\in}</code> everywhere fixes it.</li>
<li>Each footnote is, by default, put in its own separate HTML file. It gets fixed
by creating a <code>.make4ht</code> file that contains something like:
<code>settings_add {
tex4ht_sty_par = ",fn-in"
}</code>
This tells <code>make4ht</code> to pass additional arguments (here, <code>fn-in</code>) to <code>tex4ht</code>,
which change its behavior. There are <a href="https://www.kodymirus.cz/tex4ht-doc/texfourhtOptions.html">many available
options</a>.</li>
</ul>
<p>I should probably have noticed the early warning signs. One is that the default
behavior often makes zero sense: for example, this footnote problem… who would
want footnotes in a separate HTML files when all the rest is in a single HTML
file? Why is that a reasonable thing to do?</p>
<p>Also, compilation errors don't give you a clear picture of what actually goes
wrong. LaTeX is bad at this in general, but TeX4ht is definitely worse. The
error messages are often classical LaTeX errors like <code>! Extra }, or forgotten
\endgroup</code>, but that's almost never the <em>actual</em> error, since the same file
compiles fine into PDF. So looking it up error messages online doesn't help.
Instead, I fixed those early problems by bisecting the error, or by asking the
internet how to do a certain thing.</p>
<p>Still, I'm making quick progress. I wonder things like "can I put the different
sections on different HTML pages rather than having one monolithic document" and
find out that all I need to do is pass an option to TeX4ht and it works. The
option is unbelievably badly named: to tell it "make one page per subsection",
you tell it "3", because that's three subdivision levels (chapter / section /
subsection). Yes, I really mean "3". The option has no other name. You just pass
a single number to the command line.</p>
<p>But whatever. It works. I make progress. I invest time fixing things. Surely, if
I just spend a few more hours of fixing things, I'll be done. The sunk cost
fallacy starts taking its hold on me. I don't notice a thing. </p>
<h1 id="perseverance">Perseverance</h1>
<p>I start stumbling into some issues that are more difficult to fix. The first big
one is how equations display. By default, TeX4ht converts each equation into an
image, and includes the image in the HTML file. I imagine it's pretty awful for
accessibility, and it's also really ugly. The images are low-quality, stand out
in the middle of text, zooming in or out is a visual nightmare. After some
testing, I decide that the best solution is to pass the <code>mathml</code> option to
<code>tex4ht</code>, and pass the <code>html5+mathjaxnode</code> option to <code>make4ht</code> to tell it to
post-process all of the pages with <code>mathjax-node-page</code>, which converts the
MathML equations into… prettier-looking equations I think. I don't exactly
understand how it works, but MathML alone is ugly, and this is pretty. Ship it.
This requires me to install Node.js, which, urgh, but whatever.</p>
<p>I realize only afterwards that this package is deprecated, and that TeX4ht's
<a href="https://github.com/michal-h21/make4ht">GitHub repository</a> recommends using the
<code>mjcli</code> option instead. That option isn't recognized on my machine, probably
because I don't have a recent enough version. What I have works, so I don't look
further.</p>
<p>I also start cleaning up my build process. And this is where I start noticing
some behaviors of these tools that are kind of really wrong and frustrating for
no reason.</p>
<ul>
<li>One example is the <code>-d</code> option of <code>make4ht</code>, which is supposed to tell it "put
all output files in this specific subdirectory". This option is lying to you.
The files are <em>copied</em> over to this directory, and only <em>some</em> of them. So
your working directory is still cluttered with intermediary files,
logs, and HTML files.</li>
<li>I initially thought that it would be kind of like <code>latexmk</code>, running the
compilation commands multiple times until it gets the bibliography references
right. It does not do that. You have to do it manually.</li>
<li>When you realize you didn't compile what you wanted to, pressing <code>ctrl-c</code>
doesn't seem to stop the process. It does, however, make the command-line
output hang. So you have to close the terminal and open a new one again.</li>
</ul>
<p>None of these things is a huge deal-breaker. I am still making progress. I also
fix a bunch of other problems that start looking more like real weird bugs than
understandable annoyances.</p>
<ul>
<li><code>\autoref</code> did not work. I tried pretty hard to fix it, and finally gave up
and changed all the <code>\autoref</code>s into regular <code>\ref</code>s using <code>sed</code>.</li>
<li>LaTeX expressions that are perfectly fine according to <code>pdflatex</code>, like
<code>a_\text{b}</code> or <code>a^\mycommand{b}</code> (where <code>\mycommand</code> is a custom command),
failed to compile. This could be fixed by adding brackets: <code>a_{\text{b}}</code>
works, as does <code>$^{\mycommand{b}}</code>. Alas, fixing all compilation problems
isn't enough: simple expressions like <code>e^\eps</code>, where <code>\eps</code> is defined as
simply an alias of <code>\varepsilon</code>, <em>compile</em> fine, but <em>display</em> incorrectly,
so they must also be detected and changed to e.g. <code>e^{\eps}</code>.</li>
<li>But wait, this gets even worse: expressions like <code>e^{\eps}</code> are fine in text,
but if they are put in <em>macros</em>, then they no longer work. Sometimes. To solve
that final problem, I replaced all <code>_</code> and <code>^</code> in my macros by <code>\sb</code> and
<code>\sp</code>. Gross.</li>
<li>The itemized list of <code>tablenotes</code> in <code>threeparttable</code> environments did not
correctly put line breaks between items. You have to add line breaks manually.</li>
<li>Speaking of tables, <code>multirow</code> doesn't work. A workaround is to use <code>\newline</code>
within cells. There is probably a better option.</li>
<li>Having multiple <code>\tikzpicture</code> commands in a single figure resulted in really
weird visual bugs, without a compilation error: only a single picture being
shown, random text in absurd places. Putting each <code>\tikzpicture</code> in its own
cell in a <code>tabular</code> environment is a quick workaround. There is probably a
better option (<code>subfigure</code> with the right arguments maybe?).</li>
<li><code>\hat{D}</code> looked reasonable, <code>\hat{O}</code> displayed like the french Ô in
equations. Whyyyy. I fixed it by using <code>\hat{{O}}</code>. No clue why it works nor
why it happened in the first place.</li>
<li>Regular parentheses in equations are automatically sized to the biggest thing
on the same line. So if you have an equation like
<span class="math">\(f(x)=\frac{tallthing}{alsotallthing}\)</span>, the parenthesis around the <span class="math">\(x\)</span> are
comically large. You need to replace all these by <code>\left(x\right)</code> to get the
correct behavior.</li>
<li>Having a <code>cases*</code> environment nested inside of an <code>align*</code> environment failed
to compile. Replacing the <code>align*</code> environment by <code>\[ … \]</code> compiles, but the
line breaks within the <code>cases*</code> environment are ignored. I solved it by using
a <code>matrix*</code> environment instead (with the <code>[l]</code> option for correct alignment),
surrounded by <code>\left\{...\right.</code> to emulate the big left bracket.</li>
</ul>
<p>This is where I started doing some really ugly things to get around such bugs.
Using <code>grep</code> and <code>sed</code> to do large-scale changes, or doing gross things like
replacing horizontal spaces by non-breaking spaces, became routine. At that
point though, I was in too deep to reconsider my choices. So I kept going, even
as the bugs got progressively more arcane.</p>
<h1 id="pain">Pain</h1>
<p>The serious problems happened as I was trying to figure out how to get the table
of contents working as expected. It seemed to be truncated for no reason, with
very weird errors on the command-line, referencing some intermediary files. I
bisected it to a <code>%</code> symbol in the caption of a table. You read that right: I had
a correctly-escaped <code>%</code> in the legend of one of my figures, it compiled and
displayed perfectly fine, but it broke the regular table of contents. Not the
"list of figures", mind you! I didn't even have a list of figures!</p>
<p>Another problem was with chapter- or section-specific tables of content, which
are a good thing to have when everything is separated across many HTML pages.
Sadly, they sometimes had the wrong sections or subsections in them; Section 4.2
would have a few subsections from Section 4.3 in its table of contents. I tried
for a while to make an minimal working example to figure out where the problem
came from, the behavior didn't look very deterministic, so I gave up and simply
removed these altogether.</p>
<p>Captions also have their share of bizarre, non-deterministic bugs. For example,
using a formula like <code>\left[a\middle|b\right]</code> inside of a caption made
compilation fail. Removing the <code>\middle</code> part, which does not cause any issue
anywhere else, fixes it. Except that macros also sometimes fail to display the
desired formula inside captions, with e.g. a subscript being ignored. But the
exact same code without a macro would work fine, or the same macro outside a
<code>\caption{}</code> would also work fine. Bizarre stuff.</p>
<p>Eventually, I stopped trying to fix the bugs, and simply learned to work around
them, by either removing the thing entirely, or post-processing the output. This
happened, uh, a number of times.</p>
<ul>
<li>Using <code>\intertext</code> between lines of an <code>align*</code> equation, a trick to keep
equations aligned even when you put a paragraph of text between them, resulted
in the entire thing being ridiculously shifted to the right. I solved it by
changing the <code>\intertext</code> into a normal paragraph.</li>
<li>Algorithms from the <code>algorithm2e</code> package display really strangely. Removing
line numbers kind of helps, but it's not great, and the <a href="https://tex.stackexchange.com/a/292712">official
advice</a> seems to be "convert it as an
image", which, ew. I only used this environment once, so I simply converted it
into a listing.</li>
<li>The TeX4ht config file did not work as expected. Internet tells me adding
lines starting with <code>\Configure{@HEAD}</code> is supposed to add corresponding lines
in the <code><head></code> element of the generated HTML files, and you add multiple such
lines to add multiple elements. There are plenty of examples online of this
pattern being used. Somehow, on my machine, only the first such command was
added to <code><head></code>, the others appeared in the <code><body></code> instead (which, of
course, does not have the expected semantics). After a few hours trying to
debug this, I trashed that whole idea and, instead, made a Python script that
replaced the beginning and the end of each HTML page entirely instead.</li>
<li>A series of underscores got added after some of the citations at the beginning
of each chapter. I added a few lines to my Python script to get rid of them
without even trying to understand where that particular weirdness came from.</li>
</ul>
<p>The CSS part of this whole build process is also broken in interesting ways.
Two style files are generated: a main one that I think is part of TeX4ht, and
another one added by <code>mathjax-page-node</code>.</p>
<ul>
<li>The main CSS has the same commands repeated many times for no reason. It also
has styles that are "obviously wrong": <code>class="indent"</code> ends up <em>disabling</em>
the text indentation, while there are elements with the "noindent" class,
which aren't defined anywhere on CSS, so inherit the global behavior (which is
"add an indentation" on my website).</li>
<li>The mathjax CSS is fine, but the build process copies it over to the output
directory <em>every time a file is generated</em>. But when the file doesn't contain
any equation, that CSS is empty instead! So if that's the case for the <em>last</em>
file generated by the build process, its empty CSS file overwrites the correct
CSS file and all of a sudden, the equations looks terrible. I fixed it by
manually adding the "right" CSS in a fixed place.</li>
</ul>
<h1 id="futility">Futility</h1>
<p>So, it's <a href="/thesis">done</a>. I'm pretty happy about how it looks. The entire
exercise was entirely futile, of course: it's not like anyone will, y'know,
actually <em>read</em> the damn thing. But I'm weirdly glad it exists.</p>
<p>Obviously though, I'm not at all impressed by the road that was needed to get to
this point. It's infuriating that doing something like this was so hard. LaTeX
is the main way scientific research gets written up. HTML might be the main
format used by pretty much everyone on the planet to consume written content.
Why is converting one to to the other not a solved problem?</p>
<p>Of course, we know why. Incentives in academia are irremediably broken, so we're
stuck with old practices, bad formats, a lack of funds for projects that would
make everyone's life better, and a structural impossibility to do much about it.
My friend a3nm <a href="https://a3nm.net/work/research/wrong/">lays out all of these root
causes</a> much better than I possibly
could, and this LaTeX-to-HTML story is a good illustration. Imagine that we
lived in a world where it was <em>trivial</em> to make beautiful web pages out of
scientific papers. Wouldn't that encourage more researchers to share their work
more widely? Wouldn't that create whole new categories of readership, given that
most people consume content on their phone? If HTML was the default format for
research, would more people realize how <em>ridiculous</em> it is that paywalled
research papers are still a thing in 2021?</p>
<p>Anyway. I'm complaining, but I still want to finish off on a positive note: the
people who are actually doing the work of building and maintaining this tooling
are <em>heroes</em>. The many bugs and annoyances I complained about should in no way
be interpreted as a criticism of the authors of the software. Converting LaTeX
to HTML is absurdly hard because LaTeX was never designed for such a thing,
because the input language is forever stuck in the 80's, and because the
complexity of the package ecosystem is out of control. The more you dive into
how these converters work, the more you realize that the fact that they work at
all is actually pretty impressive! Massive respect to folks like <a href="https://tex.stackexchange.com/users/2891/michal-h21">Michal
Hoftich</a>, who are creating
software that solves a fundamentally difficult problem <em>and</em> spending massive
amounts of time and energy answering people's questions. Genuinely inspiring. </p>
<p>I hope that some day, that kind of work can be properly funded and rewarded. I
don't really know how we get there.</p>
<h1 id="additional-thoughts-added-in-december-2023">Additional thoughts (added in December 2023)</h1>
<p>Time has passed since I originally wrote this blog post, and a few things
happened since.</p>
<ul>
<li>I presented my thesis and this blog post as an exhibit at a workshop called
<a href="https://rethinkingmlpapers.github.io/papers/">Rethinking ML Papers</a>. I recorded a short talk about it, and a
recording is now available <a href="https://www.youtube.com/watch?v=vznwJU-WdUo">on YouTube</a>.</li>
<li><a href="https://kwarc.info/people/dginev/">Deyan Ginev</a>, one of the maintainers of <a href="https://dlmf.nist.gov/LaTeXML/">LaTeXML</a>, reached out to me
to tell me that they've landed patches to avoid fatal errors during
conversion. Versions of LaTeXML from 2022 onwards now produce a partial output
when ran on the original sources of my PhD thesis. He's now involved in the
<a href="https://ar5iv.labs.arxiv.org/">ar5iv</a> project, whose goal is to convert all papers on <a href="arxiv.org">arXiv</a>
into HTML. Super cool progress!</li>
<li><a href="https://bdtechconcepts.com/">Brian Dunn</a>, the main author of <a href="https://ctan.org/pkg/lwarp">lwarp</a>, reached out to me to ask for
the original sources of my PhD thesis, and understand more about what problems
I originally encountered. He then <em>fixed all the issues in lwarp</em> until my
thesis compiled without issue (on v0.897 and above). How impressive is this!
My original blog post said that the folks who build & maintain conversion
software are heroes, and I could not have been more right.</li>
</ul>
<p>This made me want to understand more about the technical complexity of this kind
of work, and get a better overview of the different tools that are out there
(something I wish I'd done at the <em>beginning</em> of this project rather than at the
end). I found some good discussions available online, for example
<a href="https://tug.org/pipermail/texhax/2016-March/022231.html">here</a> or
<a href="https://tug.org/pipermail/tex4ht/2016q1/001399.html">there</a>. It made me realize
that there are some profound differences with how different tools tackle the
problem.</p>
<p>If I had to do this again, I would probably use <a href="https://ctan.org/pkg/lwarp">lwarp</a>. I like its
straightforward technical approach: it uses LaTeX itself to parse the source
files and directly generate HTML. This means it can't handle unknown LaTeX
packages that implement brand new things… but also that it's less likely to lead
to super arcane errors, and that adding support for new packages is easier. It
also tries to suggest alternative packages when it encounters an unsupported
one, allowing users to solve the error at the source. I would also be cautiously
optimistic about my ability to patch lwarp itself if necessary.</p>
<p>If you've had some experience doing big LaTeX-to-HTML conversion projects like
the one described in this blog, let me know!</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>The magic of Gaussian noise2020-11-15T00:00:00+01:002021-03-17T00:00:00+01:00Damien Desfontainestag:desfontain.es,2020-11-15:/privacy/gaussian-noise.html<p>Why is Gaussian noise a popular choice to make statistics and machine learning models differentially private?</p><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p></small></p>
<hr>
<p><span class='lettrine'>P</span><strong>reviously</strong>, we used Gaussian noise to explain
the <a href="privacy-loss-random-variable.html">real meaning of <span class="math">\(\delta\)</span></a> in <span class="math">\((\varepsilon,\delta)\)</span>-differential
privacy. One question was left unanswered: why would anyone use Gaussian noise
in the first place? The guarantees it provides aren't as strong: it gives
<span class="math">\((\varepsilon,\delta)\)</span>-DP with <span class="math">\(\delta>0\)</span>, while Laplace noise provides pure
<span class="math">\(\varepsilon\)</span>-DP. This blog post gives an answer to this question, and describes
the situations in which Gaussian noise excels.</p>
<h1 id="gaussian-noise-is-nice">Gaussian noise is nice</h1>
<p>A first advantage of Gaussian noise is that the distribution itself behaves
nicely. It's called the <a href="https://en.wikipedia.org/wiki/Normal_distribution"><em>normal</em> distribution</a> for a reason: it has
convenient properties, and is very widely used in natural and social sciences.
People often use it to model random variables whose actual distribution is
unknown. If you sum many independent random variables, you <a href="https://en.wikipedia.org/wiki/Central_limit_theorem">end up with a normal
distribution</a>. And these are just a few of the <a href="https://en.wikipedia.org/wiki/Normal_distribution#Properties">many
properties</a> of this fundamental distribution. Thus, most data
analysts and scientists are already familiar with Gaussian noise. It's
convenient when you release anonymized statistics: analysts don't need to learn
too many new concepts to understand what you're doing to protect the data.</p>
<p>A second advantage is that the Gaussian distribution has nice, <em>thin</em> tails. The
vast majority of its probability mass is focused around its mean. Take a normal
distribution with mean 0 and standard deviation <span class="math">\(\sigma\)</span>. The <a href="https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule">68–95–99.7
rule</a> says that a random variable sampled from this distribution will be:</p>
<ul>
<li>in <span class="math">\([-\sigma,\sigma]\)</span> with 68% probability;</li>
<li>in <span class="math">\([-2\sigma,2\sigma]\)</span> with 95% probability;</li>
<li>and in <span class="math">\([-3\sigma,3\sigma]\)</span> with 99.7% probability.</li>
</ul>
<p>It even gets better as you go further away from the mean. The probability that
the random variable is <em>outside</em> <span class="math">\([-k\sigma,k\sigma]\)</span> decreases faster than
<span class="math">\(e^{-k^2/2}\)</span>. In practice, you're rarely surprised by the values that a Gaussian
distribution takes. Even if you sample <span class="math">\(1,000,000\)</span> values, they are all probably
going to be within <span class="math">\([-5\sigma,5\sigma]\)</span>.</p>
<p>Laplace, by comparison, isn't quite as nice. Its tails decrease exponentially fast,
but that's still much slower than Gaussian tails. Suppose you sample <span class="math">\(1,000,000\)</span>
values from a Laplace distribution of standard deviation <span class="math">\(\sigma\)</span>. On average,
<strong>849</strong> of them will be outside <span class="math">\([-5\sigma,5\sigma]\)</span>.</p>
<p>OK, so Gaussian noise is nice. But that does not change a simple fact: to get
a comparable level of privacy for a single statistic, Laplace is much better.
Assume that we're adding noise to a simple count, of sensitivity <span class="math">\(1\)</span>. This graph
compares the Laplace noise needed to get <span class="math">\(\varepsilon=1\)</span>, and Gaussian noise
needed to get <span class="math">\(\varepsilon=1\)</span> and <span class="math">\(\delta=10^{-5}\)</span>.</p>
<p><center>
<img alt="Graph showing a Laplace distribution with scale 1/ln(3) and a Gaussian distribution of standard deviation 3.426, both centered on 0" src="https://desfontain.es/privacy/images/laplace-gaussian-ln-3.svg">
</center></p>
<p>Despite its weaker privacy guarantees, the Gaussian distribution is much
flatter. Its standard deviation is over 3.4, while Laplace's is less than 1.3.
Thus, much more noise will need to be added, and analysts care a lot about
minimizing the noise. Why, then, would Gaussian noise be a good option? The answer
is simple: because it gets better when you're publishing a lot of statistics.</p>
<h1 id="from-one-to-many-statistics">From one to many statistics</h1>
<p>In most of our <a href="differential-privacy-in-practice.html">previous examples</a>, we assumed that each individual
appeared in a <em>single</em> statistic. This case is common, for example when
partitioning people based on demographic information. But in many applications,
this assumption does not hold. Imagine, for example, that you want to answer the
question: « what types of specialized physicians did people visit in the past 10
years? »</p>
<p>Assume we're working in the <a href="local-global-differential-privacy.html">central model</a>. We have a dataset of
〈patient ID, specialist type〉 pairs, and each record corresponds to an
individual visiting a specialized physician (cardiologist, dermatologist,
radiologist, etc.). We want to count the number of unique patient IDs per
specialty.</p>
<p>Note that each patient can only influence a single count <em>once</em>. We count
<em>distinct</em> patient IDs: if you visit dermatologists 10 times, you will only add 1
to the "dermatologist" count. However, a single patient might visit many types of
specialized physicians. There are <a href="https://en.wikipedia.org/wiki/Medical_specialty">many kinds of
specialties</a>, and a single
patient might influence all the counts. Say there are 50 of them.</p>
<p>How to make these counts differentially private? A first solution is to <a href="differential-privacy-in-practice.html#multiple-statistics"><em>split
the privacy budget</em></a> across all the counts. Here, we can
split our privacy budget in 50. If we want to achieve <span class="math">\(\varepsilon=1\)</span>, we
compute <span class="math">\(\varepsilon'=1/50=0.02\)</span>, and add Laplace noise of scale
<span class="math">\(1/\varepsilon'=50\)</span> to all the counts.</p>
<p>Unfortunately, this is a lot of noise.</p>
<p><center>
<img alt="Graph showing a Laplace distribution with scale 0.02, centered on 0" src="https://desfontain.es/privacy/images/laplace-0.02.svg">
</center></p>
<p>Fortunately, this is exactly the kind of situation in which Gaussian noise
shines. When a single patient can impact <span class="math">\(k\)</span> distinct statistics, we need to
scale Laplace noise by <span class="math">\(k\)</span>. By contrast, Gaussian noise must only be scaled by
<span class="math">\(\sqrt{k}\)</span>. Comparing the two gives a much more flattering view of the power of
Gaussian noise.</p>
<p><center>
<img alt="Graph showing a Laplace distribution with scale 0.02, and a Gaussian distribution of standard deviation 26.38, both centered on 0" src="https://desfontain.es/privacy/images/gaussian-laplace-50.svg">
</center></p>
<p>OK, so that's the general idea. Now, <em>why</em> does that happen? How come
composition doesn't seem to behave in the same way for Laplace and Gaussian? To
understand this better, we'll first introduce the concept of <em>sensitivity</em>.</p>
<h1 id="different-kinds-of-sensitivities">Different kinds of sensitivities</h1>
<p>Consider our example above. For each type of specialized physician, we count the
people who consulted with one. But we won't consider this histogram as 50
different counting queries. Instead, we'll consider it as a <em>single</em> function,
with values in <span class="math">\(\mathbb{N}^{50}\)</span>. It outputs a <em>vector</em>: a list of 50
coordinates, each of which corresponds to a fixed specialty. How to make such a
function <span class="math">\(f\)</span> differentially private? We'll add noise, scaled by the function's
<em>sensitivity</em>.</p>
<p>The sensitivity of a function measures how much its output can change when you
add one record in the database. If the function returns a single number, we
measure the <em>absolute value</em> of the difference: the sensitivity of <span class="math">\(f\)</span> is the
maximum value of <span class="math">\(\left|f\left(D_1\right)-f\left(D_2\right)\right|\)</span>. We already
encountered the sensitivity before: when <a href="differential-privacy-in-practice.html#counting-things">counting things</a>, if each
patient can change the statistic by more than <span class="math">\(1\)</span>, we needed to scale the noise
accordingly. The same happened for sums.</p>
<p>Here, the function <span class="math">\(f\)</span> returns a vector. How do we measure the distance between
two vectors? We have a few options. We could use the <a href="https://en.wikipedia.org/wiki/Taxicab_geometry">Manhattan
distance</a>, or the <a href="https://en.wikipedia.org/wiki/Euclidean_distance">Euclidean distance</a>, or even <a href="https://en.wikipedia.org/wiki/Norm_(mathematics)#p-norm">weirder
stuff</a>. As it turns out, the distance we need to use depends on which noise
function we're adding. Laplace noise is scaled by the <span class="math">\(L^1\)</span> sensitivity, itself
based on the Manhattan distance. Here is its definition, denoting by <span class="math">\(f_i(D)\)</span>
the <span class="math">\(i\)</span>-th coordinate of <span class="math">\(f(D)\)</span>:</p>
<div class="math">$$
\Delta_1(f) = \max \sum_{i=1}^{50} \left|f_i\left(D_1\right)-f_i\left(D_2\right)\right|
$$</div>
<p style="text-indent: 0em">where the <span class="math">\(\max\)</span> is taken over <span class="math">\(D_1\)</span>
and <span class="math">\(D_2\)</span> differing in a single record. This is easy to understand: you just sum
the sensitivities for each coordinate. For our function <span class="math">\(f\)</span>, we have
<span class="math">\(\Delta_1(f)=50\)</span>: Laplace noise needs to be scaled by 50. You might have noticed
that this is equivalent to using simple composition: the scale of Laplace noise
is <span class="math">\(\Delta_1/\varepsilon\)</span>, so dividing <span class="math">\(\varepsilon\)</span> by <span class="math">\(50\)</span> is the same as
considering all coordinates together.</p>
<p>By contrast, Gaussian noise needs to be scaled by the <span class="math">\(L^2\)</span> sensitivity. This
type of sensitivity is based on the Euclidean distance, and defined by:</p>
<div class="math">$$
\Delta_2(f) = \max \sqrt{\sum_{i=1}^{50} {\left|f_i\left(D_1\right)-f_i\left(D_2\right)\right|}^2}
$$</div>
<p style="text-indent: 0em">still taking the <span class="math">\(\max\)</span> over <span class="math">\(D_1\)</span> and
<span class="math">\(D_2\)</span> differing in a single record. This formula might look complicated, but the
Euclidean distance is a simple concept: it's the length of a straight line
between two points. If you only have two dimensions, this formula might be
reminiscent of the <a href="https://en.wikipedia.org/wiki/Pythagorean_theorem">Pythagorean theorem</a>!</p>
<p>The standard deviation <span class="math">\(\sigma\)</span> of Gaussian noise will be proportional to
<span class="math">\(\Delta_2(f)\)</span>. Let's compute this value for our function. Each patient can
change a single count by at most one, and each can change all counts. Thus:</p>
<div class="math">$$
\Delta_2(f) = \sqrt{\sum_{i=1}^{50} 1^2} = \sqrt{50} \approx 7.07.
$$</div>
<p>The noise scales with the <em>square root</em> of the number of counts. This is key to
Gaussian's superiority in such situations: the <span class="math">\(L^2\)</span> sensitivity grows much more
slowly than the <span class="math">\(L^1\)</span> sensitivity. As a result, scaling the noise by the
sensitivity hurts accuracy much less. Of course, using Gaussian noise gives you
<span class="math">\((\varepsilon,\delta)\)</span>-DP, not pure DP, so there is still a tradeoff. As we saw
in the <a href="privacy-loss-random-variable.html">previous article</a>, this isn't a super scary <span class="math">\(\delta\)</span>, so it's
generally worth it.</p>
<p><a name="formula"></a></p>
<p><a href="https://arxiv.org/abs/1805.06530">This paper</a> (Theorem 8) gives the exact
formula for calibrating Gaussian noise depending on <span class="math">\(\Delta_2(f)\)</span>, <span class="math">\(\varepsilon\)</span>
and <span class="math">\(\delta\)</span>. You need to pick <span class="math">\(\sigma\)</span> such that the following equality holds:</p>
<div class="math">$$
g\left(\frac{\Delta_2(f)}{\sigma},\varepsilon\right) = \delta
$$</div>
<p style="text-indent: 0em">where <span class="math">\(g\)</span> is a complicated function. As
you can see, increasing <span class="math">\(\Delta_2(f)\)</span> and <span class="math">\(\sigma\)</span> simultaneously has no effect:
<span class="math">\(\sigma\)</span> is proportional to <span class="math">\(\Delta_2(f)\)</span>. There is no analytic form for
<span class="math">\(\sigma\)</span>, but since <span class="math">\(g\)</span> is monotonic, you use e.g. a binary search to
approximate its value. If you want to know the exact formula, click here:
<button id="toggleMath"></button>.</p>
<div id="math" style="display: none; border-left: double; padding-left: 10px">
<p>OK. You need to satisfy the inequality:</p>
<p>
<div class="math">$$
\Phi\left(\frac{\Delta_2(f)}{2\sigma}-\frac{\varepsilon\sigma}{\Delta_2(f)}\right)
- e^\varepsilon \Phi\left(-\frac{\Delta_2(f)}{2\sigma}-\frac{\varepsilon\sigma}{\Delta_2(f)}\right)
\le \delta
$$</div>
</p>
<p style="text-indent: 0em">where <span class="math">\(\Phi\)</span> is the <a href="https://en.wikipedia.org/wiki/Normal_distribution#Cumulative_distribution_function">cumulative
distribution function</a> of a Gaussian distribution of mean <span class="math">\(0\)</span> and variance
<span class="math">\(1\)</span>. This is an inequality, but in practice, you want to get as close as
possible to equality, to add as little noise as possible. Implementing this
correctly on floating-point machines is, as usual, pretty tricky, <a href="https://github.com/google/differential-privacy/blob/0e95b99af5fedff95b8cab3739009bbc1366c068/go/noise/gaussian_noise.go#L300">here's an
example</a>.</p>
<p>This formula, called the <em>Analytic Gaussian mechanism</em>, is different from the
more famous Gaussian mechanism introduced in <a href="https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf">this paper</a>
(Appendix A). The analytic version is tighter, and more importantly, it works
for arbitrary values of <span class="math">\(\varepsilon\)</span>. The formula in the original Gaussian
mechanism, by contrast, only works for <span class="math">\(\varepsilon<1\)</span>.</p>
</div>
<p>Now, why do these two noise distributions work so differently? Rather than
proving this formally, here is a visual intuition. Let's look at the density
function of Gaussian and Laplace noise, in two dimensions. The first is
Gaussian, the second is Laplace.</p>
<p><center>
<img alt="Graph showing a two-dimensional Gaussian distribution" src="https://desfontain.es/privacy/images/gaussian-2d.svg">
<img alt="Graph showing a two-dimensional Laplace distribution" src="https://desfontain.es/privacy/images/laplace-2d.svg">
</center></p>
<p>As you can see, Gaussian noise has a circular shape, while Laplace noise has a
square shape. How indistinguishable are two points, when noise is added to both
of them? With Gaussian noise, it depends on their distance from each other <em>in
a straight line</em>. By contrast, with Laplace, it depends on how far they are in
<a href="https://en.wikipedia.org/wiki/Taxicab_geometry">Manhattan distance</a>.</p>
<p>In conclusion, whether to use Laplace or Gaussian noise depends on two things:</p>
<ul>
<li>whether we are OK with a non-zero <span class="math">\(\delta\)</span>;</li>
<li>how many statistics a single individual can influence.</li>
</ul>
<p>The first point is clear: if we want <span class="math">\(\delta=0\)</span>, we can't use Gaussian noise.
Let's quantify the second point. If a single person can impact at most one
statistic, Laplace is better. If they can impact many, Gaussian is better. Where
does the boundary lie? The following graph answers this question, comparing both
mechanisms by their standard deviation. We pick <span class="math">\(\varepsilon=1\)</span> and
<span class="math">\(\delta=10^{-5}\)</span>, and we assume that each person can influence each statistic at
most once.</p>
<p><center>
<img alt="Graph comparing the standard deviation of the noise" src="https://desfontain.es/privacy/images/laplace-gaussian-comparison.svg">
</center></p>
<p>For these values of <span class="math">\(\varepsilon\)</span> and <span class="math">\(\delta\)</span>, Gaussian noise is better if each
individual can influence 8 statistics or more. Of course, with different privacy
parameters, the result might differ. But as the impact of a single individual
grows, Gaussian noise will always end up being the least noisy choice.</p>
<h1 id="further-uses">Further uses</h1>
<p>What if each person can influence each statistic <em>differently</em>? Suppose, for
example, that we count the number of <em>visits</em> to each type of physician. A
single patient can add many visits to a single count. Worse, the maximum number
of visits per patient can vary across physician types. Can we still use Gaussian
noise? The answer (hat tip to <a href="https://crypto.stackexchange.com/q/85581">Mark</a>) is <em>yes</em>: you can scale down
each statistic so the sensitivity becomes <span class="math">\(1\)</span>, add noise to them, then scale
them back up. This makes Gaussian noise even more powerful: if you compute many
statistics on the same data, you can use the Gaussian mechanism to reduce the
noise magnitude, even if the statistics are completely unrelated.</p>
<p>Finally, Gaussian noise is heavily used in differentially private machine
learning. The fundamental reason is the same. Consider methods like <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent">stochastic
gradient descent</a>, a popular algorithm in machine learning. At each
iteration of this method, each data point influences many coordinates of a
vector. To make it differentially private, we need to add noise to all
coordinates. Thus, Gaussian noise is a good choice, for the exact same reason.
Machine learning is full of more Gaussian-related goodness, but this article is
long enough already.</p>
<p>Maybe we'll come back to it in future posts! The <a href="more-useful-results-dp.html">next article</a>, though,
tackles a much simpler problem: what do you do when you try using differential
privacy on your data, but the results aren't accurate enough?</p>
<hr>
<p><small>Thanks to [Frank McSherry] and [Antoine Amarilli] for their helpful
comments.</small></p>
<script type="text/javascript">
var button = document.getElementById('toggleMath');
var defaultButton = 'show me the scary math';
button.innerHTML = defaultButton
button.addEventListener('click', function (event) {
button.innerHTML = button.innerHTML == defaultButton ? 'hide the scary math' : defaultButton;
math = document.getElementById('math');
math.style.display = math.style.display == 'none' ? 'block' : 'none';
});
</script>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>The privacy loss random variable2020-03-06T00:00:00+01:002022-07-23T00:00:00+02:00Damien Desfontainestag:desfontain.es,2020-03-06:/privacy/privacy-loss-random-variable.html<p>What does <span class="math">\(\delta\)</span> <em>really</em> mean in <span class="math">\((\varepsilon,\delta)\)</span>-differential privacy? Let's explain this using a central concept: the privacy loss random variable.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script><script type='text/javascript'>if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p></small></p>
<hr>
<p><span class='lettrine'>R</span><strong>emember</strong> the notion of « <a href="almost-differential-privacy.html">almost</a> »
differential privacy? We changed the <a href="differential-privacy-in-more-detail.html">original</a> definition to add a new
parameter, <span class="math">\(\delta\)</span>. We said that <span class="math">\(\delta\)</span> was « the probability that something
goes wrong ». This was a bit of a shortcut: this nice and easy intuition is
sometimes not exactly accurate. In this post, I'll do two things. I'll introduce
a crucial concept in differential privacy: the « privacy loss random variable ».
Then, I'll use it to explain what <span class="math">\(\delta\)</span> <em>really</em> means.</p>
<p>Friendly heads-up: this post has slightly more math than the rest of this
series. But don't worry! I made it as nice and visual as I could, with graphs
instead of equations. All the equations are in a proof hidden by default.</p>
<h1 id="the-privacy-loss-random-variable">The privacy loss random variable</h1>
<p>Recall the setting of the definition of <span class="math">\(\varepsilon\)</span>-DP (short for differential
privacy). The attacker tries to distinguish between two databases <span class="math">\(D_1\)</span> and
<span class="math">\(D_2\)</span>, that differ by only one record. If a mechanism <span class="math">\(A\)</span> is <span class="math">\(\varepsilon\)</span>-DP,
then <span class="math">\(A\left(D_1\right)\)</span> and <span class="math">\(A\left(D_2\right)\)</span> will return output <span class="math">\(O\)</span> with
similar probability:</p>
<div class="math">$$
\mathbb{P}[A(D_1)=O] \le e^\varepsilon\cdot\mathbb{P}[A(D_2)=O].
$$</div>
<p>The equality also goes in the other direction, but the relation between <span class="math">\(D_1\)</span>
and <span class="math">\(D_2\)</span> is symmetrical, so we only use this one inequality, to simplify.</p>
<p>We said before that the <span class="math">\(\varepsilon\)</span> in <span class="math">\(\varepsilon\)</span>-DP was the <em><a href="differential-privacy-awesomeness.html#quantify">maximal
knowledge gain</a></em> of the attacker. We defined this knowledge gain in <a href="differential-privacy-in-more-detail.html#quantifying">Bayesian
terms</a>, where the attacker is trying to guess if the real database <span class="math">\(D\)</span> is <span class="math">\(D_1\)</span>
and <span class="math">\(D_2\)</span>. We saw that <span class="math">\(\varepsilon\)</span> bounds the evolution of betting odds. For
each <span class="math">\(O\)</span>, we had:</p>
<div class="math">$$
\frac{\mathbb{P}\left[D=D_1\mid A(D)=O\right]}{\mathbb{P}\left[D=D_2\mid A(D)=O\right]} \le
e^\varepsilon\cdot\frac{\mathbb{P}\left[D=D_1\right]}{\mathbb{P}\left[D=D_2\right]}
$$</div>
<p>What if we don't just want to bound this quantity, but calculate it for a given
output <span class="math">\(O\)</span>? Let us define:</p>
<div class="math">$$
\mathcal{L}_{D_1,D_2}(O)
= \ln\frac{
\frac{\mathbb{P}\left[D=D_1\mid A(D)=O\right]}{\mathbb{P}\left[D=D_2\mid A(D)=O\right]}
}{
\frac{\mathbb{P}\left[D=D_1\right]}{\mathbb{P}\left[D=D_2\right]}.
}
$$</div>
<p>This formula looks scary, but the intuition behind it is pretty simple. The
denominator corresponds to the <em>initial</em> betting odds for <span class="math">\(D_1\)</span> vs. <span class="math">\(D_2\)</span>. How
likely is one option vs. the other, before looking at the result of the
mechanism. In Bayesian terms, this is called the "prior". Meanwhile, the
numerator of the fraction is the betting odds <em>afterwards</em> — the "posterior".
Differential privacy guarantees that <span class="math">\(\mathcal{L}_{D_1,D_2}(O)\le\varepsilon\)</span>
for all <span class="math">\(O\)</span>.</p>
<p><a href="https://arbital.com/p/bayes_rule/?l=1zq">Bayes' rule</a> allows us to reformulate
this quantity:</p>
<div class="math">$$
\mathcal{L}_{D_1,D_2}(O)
=
\ln\left(\frac{\mathbb{P}\left[A(D_1)=O\right]}{\mathbb{P}\left[A(D_2)=O\right]}\right).
$$</div>
<p>This is called the <em>privacy loss random variable</em> (PLRV for short).
<strong>Intuitively, the PLRV is the « actual <span class="math">\(\varepsilon\)</span> value » for a specific
output <span class="math">\(O\)</span>.</strong> Why is it a random variable? Because typically, we consider
<span class="math">\(\mathcal{L}_{D_1,D_2}(O)\)</span> when <span class="math">\(O\)</span> varies according to <span class="math">\(A(D_1)\)</span>, which we
assume is the "real" database.</p>
<p>OK, this is very abstract. We need a concrete example.</p>
<h1 id="a-concrete-example">A concrete example</h1>
<p>Suppose that we're counting the number of people with blue eyes in the dataset.
We make this diferentially private by adding <a href="differential-privacy-in-practice.html">Laplace noise</a> of scale
<span class="math">\(1/\ln(3)\)</span>, to get <span class="math">\(\varepsilon=\ln(3)\)</span>. The attacker hesitates between two
possible datasets: one with <span class="math">\(1000\)</span> blue-eyed people, the other with <span class="math">\(1001\)</span>. The
<em>real</em> number is <span class="math">\(1000\)</span>, but the attacker doesn't know that. The two
distributions look like this:</p>
<p><center>
<img alt="Graph showing two Laplace distributions with scale 1/ln(3), centered on 1000 and 1001" src="https://desfontain.es/privacy/images/two-laplace-ln-3.svg">
</center> </p>
<p>Let's consider three possible outputs of the mechanism, given the "real"
database is <span class="math">\(D_1\)</span>. We represent them below as <span class="math">\(O_1\)</span>, <span class="math">\(O_2\)</span>, and <span class="math">\(O_3\)</span>.</p>
<p><center>
<img alt="Graph showing the previous Laplace distributions, with three points O1, O2 and O3 marked respectively at x=999, x=1000.5 and x=1003" src="https://desfontain.es/privacy/images/two-laplace-ln-3-three-points.svg">
</center> </p>
<p>Say the attacker is very uncertain: initially, they give equal probabilities to
<span class="math">\(D_1\)</span> and <span class="math">\(D_2\)</span>. What are they going to think once we give them the output of
the mechanism?</p>
<ul>
<li>If we return <span class="math">\(O_1\)</span>, the attacker is starting to suspect that the real database
is <span class="math">\(D_1\)</span>. There's a larger chance to get that output if <span class="math">\(D=D_1\)</span> than if
<span class="math">\(D=D_2\)</span>. How much larger? Exactly 3 times larger: the attacker's knowledge is
tripled.</li>
<li>If we return <span class="math">\(O_2\)</span>, the attacker is like: ¯\_(ツ)_/¯. This is not giving
them much information. This output could have come from <span class="math">\(D_1\)</span>, but it could
just as well have come from <span class="math">\(D_2\)</span>. The attacker's knowledge doesn't change.</li>
<li>If we return <span class="math">\(O_3\)</span>, the attacker is getting <em>tricked</em> with wrong information.
They will think it's more likely that the real database is <span class="math">\(D_2\)</span>. Their
"knowledge" is divided by 3.</li>
</ul>
<p>Let's look at all possible events <span class="math">\(O=A(D_1)\)</span>, and <em>order</em> them. We'll put the
ones that help the attacker most first, and look at the value of
<span class="math">\(\mathcal{L}_{D_1,D_2}(O)\)</span>. Let's call this <span class="math">\(\mathcal{L}\)</span>, for short, and plot
it.</p>
<p><center>
<img alt="Graph showing the PLRV for the Laplace distribution depending on the output" src="https://desfontain.es/privacy/images/plrv-laplace.svg">
</center> </p>
<p>This is why Laplace noise is so nice: look at this neat horizontal line. Oh my
god. It even has a straight diagonal. It never goes above
<span class="math">\(\varepsilon\approx1.1\)</span>: a beautiful visual proof that Laplace noise gives
<span class="math">\(\varepsilon\)</span>-DP.</p>
<p>Let's change the graph above to more accurately represent that <span class="math">\(\mathcal{L}\)</span> is
a <em>random variable</em>. On the <span class="math">\(x\)</span>-axis, we represent all events according to their
probability. We're also more interested in <span class="math">\(\exp(\mathcal{L})\)</span>, so let's plot
that instead of <span class="math">\(\mathcal{L}\)</span>.</p>
<p><center>
<img alt="Graph showing the exponential of the PLRV for the Laplace distribution, where the x-axis represents the probability space" src="https://desfontain.es/privacy/images/exp-plrv-laplace.svg">
</center> </p>
<p>Now, what if you were using some other type of noise? Say, from a <a href="https://en.wikipedia.org/wiki/Normal_distribution">normal
distribution</a>? It would make data analysts happier: Laplace noise is weird to
them, it never shows up in the real world. Normal distributions, by contrast,
are familiar and friendly. A lot of natural data distributions can be modeled
with them.</p>
<p>In the context of differential privacy, the normal distribution is called
« Gaussian noise ». Let's try to add Gaussian noise, of variance
<span class="math">\(\sigma^2=2\)</span>:</p>
<p><center>
<img alt="Graph showing two normal distributions with variance 2, centered on 1000 and 1001" src="https://desfontain.es/privacy/images/gaussian-2.svg">
</center> </p>
<p>OK, looks reasonable, now let's see what <span class="math">\(e^\mathcal{L}\)</span> looks like:</p>
<p><center>
<img alt="Graph showing the exponential of the PLRV for the normal distribution, where the x-axis represents the probability space" src="https://desfontain.es/privacy/images/exp-plrv-gaussian.svg">
</center> </p>
<p>Ew. Look at this line going up to infinity on the left side. Gross. We can't
just draw a line at <span class="math">\(e^\varepsilon\)</span> and say "everything is underneath". What do
we do, then? We <a href="almost-differential-privacy.html">cheat</a>, and use a <span class="math">\(\delta\)</span>.</p>
<h1 id="delta-and-the-plrv"><span class="math">\(\delta\)</span> and the PLRV</h1>
<p>In a <a href="almost-differential-privacy.html">previous article</a>, we said that the <span class="math">\(\delta\)</span> in
<span class="math">\((\varepsilon,\delta)\)</span>-DP is the probability that something terrible happens.
What does that mean in the context of Gaussian noise? First, we pick an
arbitrary <span class="math">\(\varepsilon\)</span>, say, <span class="math">\(\varepsilon=\ln(3)\)</span>. Then, we look at how likely
it for <span class="math">\(e^\mathcal{L}\)</span> to be above the <span class="math">\(e^\varepsilon=3\)</span> line. It's easy to do:
the <span class="math">\(x\)</span>-axis is the probability space, so we can simply measure the width of the
bad events.</p>
<p><center>
<img alt="Same graph, but with δ marked at x=0.05, where the curve is approximately equal to 3" src="https://desfontain.es/privacy/images/exp-plrv-gaussian-delta.svg">
</center> </p>
<p>This simple intuition is correct: this mechanism is <span class="math">\((\ln(3),\delta_1)\)</span>-DP, with
<span class="math">\(\delta_1\approx0.054\)</span>. But it misses an important subtlety. Let's zoom in on
the part where things go wrong, and consider two possible outputs.</p>
<p><center>
<img alt="Same graph, zoomed on the "bad events" part before 0.05, with two points O1 and O2 marked respectively at x=0.045 and x=0.002" src="https://desfontain.es/privacy/images/exp-plrv-gaussian-two-bad-events.svg">
</center> </p>
<p>Returning <span class="math">\(O_1\)</span> is not great: <span class="math">\(e^\mathcal{L}>e^\varepsilon\)</span>. But it's not
<em>terrible</em>: the privacy loss is only a tiny bit larger than we'd hope. Returning
<span class="math">\(O_2\)</span>, however, is scary news: <span class="math">\(e^\mathcal{L}\)</span> is huge. Intuitively, <span class="math">\(O_2\)</span> leaks
much more information than <span class="math">\(O_1\)</span>.</p>
<p>With our way of quantifying <span class="math">\(\delta\)</span>, we don't account for this. We only measure
the <span class="math">\(x\)</span>-axis. What we count is <em>whether</em> <span class="math">\(e^\mathcal{L}\)</span> is above the line, not
<em>how much</em> it's above the line. For each bad event of probability <span class="math">\(p\)</span>, we're
adding <span class="math">\(p\times1\)</span> to the <span class="math">\(\delta\)</span>. A finer approach is to <em>weigh</em> the bad events
by "how bad they are". We want to give a "weight" of <span class="math">\(\approx1\)</span> to the very bad
events, and a weight of <span class="math">\(\approx0\)</span> to the "not too bad" ones.</p>
<p>To do this, we transform a bit the curve above by doing two things. First, we
take the <em>inverse</em> of the curve: very bad events are now close to <span class="math">\(0\)</span> instead of
very large. Second, we <em>normalize</em> the curve by taking the ratio
<span class="math">\(e^\varepsilon/e^\mathcal{L}\)</span>. This way, events that are "not too bad" are close
to <span class="math">\(1\)</span>.</p>
<p><center>
<img alt="Plotting exp(ε)/exp(PLRV) and highlighting the area under 1" src="https://desfontain.es/privacy/images/exp-plrv-gaussian-inversed.svg">
</center> </p>
<p>This allows us to consider the <em>area</em> between the curve and the <span class="math">\(y=1\)</span> line. When
<span class="math">\(\mathcal{L}\)</span> is very large, the inverse is close to <span class="math">\(0\)</span>, so the distance to <span class="math">\(1\)</span>
is almost 1. And when <span class="math">\(\mathcal{L}\)</span> is close to <span class="math">\(\varepsilon\)</span>, the ratio is one,
and the distance is almost 0. <em>Very bad</em> events count more than <em>sort of bad</em>
events.</p>
<p>This is the tighter, exact characterization of <span class="math">\(\delta\)</span>. In
<span class="math">\((\varepsilon,\delta)\)</span>-DP, the <span class="math">\(\delta\)</span> is the area highlighted above. It is the
mass of all possible bad events, <em>weighted</em> by how likely they are and how bad
they are. This tells us that the mechanism is <span class="math">\((\ln(3),\delta_2)\)</span>-DP with
<span class="math">\(\delta_2\approx0.011\)</span>, a much better characterization than before.</p>
<p>The typical definition of <span class="math">\((\varepsilon,\delta)\)</span>-DP doesn't use this complicated
formulation. A mechanism <span class="math">\(A\)</span> is <span class="math">\((\varepsilon,\delta)\)</span>-DP if for any neighboring
<span class="math">\(D_1\)</span> and <span class="math">\(D_2\)</span>, and any set <span class="math">\(S\)</span> of possible outputs:</p>
<div class="math">$$
\mathbb{P}[A(D_1)\in S] \le e^\varepsilon\cdot\mathbb{P}[A(D_2)\in S]+\delta.
$$</div>
<p>This definition is equivalent to the previous characterization. If you want to
see the proof of that, click here: <button id="toggleProof"></button></p>
<div id="proof" style="display: none; border-left: double; padding-left: 10px">
<p>Fix a mechanism <span class="math">\(A\)</span> and a <span class="math">\(\varepsilon\ge0\)</span>. There is a <span class="math">\(\delta\)</span> such that <span class="math">\(A\)</span>
is <span class="math">\((\varepsilon,\delta)\)</span>-DP: with <span class="math">\(\delta=1\)</span>, that's trivial and meaningless.
The interesting question is: what is the <em>smallest</em> possible <span class="math">\(\delta\)</span> such that
<span class="math">\(A\)</span> is <span class="math">\((\varepsilon,\delta)\)</span>-DP? For each possible set of outputs <span class="math">\(S\)</span>, we can
compute:</p>
<p>
<div class="math">$$
\delta_S = \mathbb{P}[A(D_1)\in S] - e^\varepsilon\cdot\mathbb{P}[A(D_2)\in S]
$$</div>
</p>
<p>The definition is satisfied iff <span class="math">\(\delta_S\le\delta\)</span> for all <span class="math">\(S\)</span>. So we have:</p>
<p>
<div class="math">$$
\delta = \max_{S} \left(\mathbb{P}[A(D_1)\in S] - e^\varepsilon\cdot\mathbb{P}[A(D_2)\in S]\right).
$$</div>
</p>
<p>It is easy to notice that you only care about the outputs <span class="math">\(O\)</span> such that:</p>
<p>
<div class="math">$$
\mathbb{P}[A(D_1)=O] > e^\varepsilon\cdot\mathbb{P}[A(D_2)=O].
$$</div>
</p>
<p>All other outputs would make <span class="math">\(\delta_\max\)</span> <em>smaller</em>, not larger. So the set <span class="math">\(S\)</span>
that maximizes the quantity above is:</p>
<p>
<div class="math">$$
S_\max = \left\{O \mid \mathbb{P}[A(D_1)=O] > e^\varepsilon\cdot\mathbb{P}[A(D_2)=O]\right\}.
$$</div>
</p>
<p>We can convert this to:</p>
<p>
<div class="math">$$
\begin{align}
\delta
& = \mathbb{P}[A(D_1)\in S_\max] - e^\varepsilon\cdot\mathbb{P}[A(D_2)\in S_\max] \\
& = \sum_{O\in S_\max} \left(\mathbb{P}[A(D_1)=O] - e^\varepsilon\cdot\mathbb{P}[A(D_2)=O]\right) \\
& = \sum_{O\in S_\max} \mathbb{P}[A(D_1)=O] \left(1 - \frac{e^\varepsilon}{e^{\mathcal{L}_{D_1,D_2}(O)}}\right).
\end{align}
$$</div>
</p>
<p>Now, instead of summing only <span class="math">\(O\in S_\max\)</span>, we could sum all possible <span class="math">\(O\)</span>, and
nullify the ones that aren't in <span class="math">\(S_\max\)</span>.</p>
<p>
<div class="math">$$
\delta
= \sum_{O} \mathbb{P}[A(D_1)=O] \max\left(0, 1 - \frac{e^\varepsilon}{e^{\mathcal{L}_{D_1,D_2}(O)}}\right).
$$</div>
</p>
<p>Now, this is an expected value:</p>
<p>
<div class="math">$$
\delta
= \mathbb{E}_{O\sim A(D_1)} \left[ \max\left(0, 1 - \frac{e^\varepsilon}{e^{\mathcal{L}_{D_1,D_2}(O)}}\right)\right]
$$</div>
</p>
<p style="text-indent: 0em">and this formula corresponds exactly to
the area between the curve above and <span class="math">\(1\)</span>.</p>
</div>
<h1 id="what-about-infinity-values">What about infinity values?</h1>
<p>Using Gaussian noise, all possible values of <span class="math">\(\mathcal{L}\)</span> are <em>finite</em>. But for
some mechanisms <span class="math">\(A\)</span>, there are outputs <span class="math">\(O\)</span> such that <span class="math">\(\mathbb{P}[A(D_1)=O]>0\)</span>,
but <span class="math">\(\mathbb{P}[A(D_2)=O]=0\)</span>. In that case, <span class="math">\(\mathcal{L}(O)=\infty\)</span>. This kind
of output is called a <em>distinguishing event</em>. If we return a distinguishing
event, the attacker immediately finds out that <span class="math">\(D\)</span> is <span class="math">\(D_1\)</span> and not <span class="math">\(D_2\)</span>. This
is the case for the "thresholding" example we looked at <a href="almost-differential-privacy.html">previously</a>.</p>
<p>Our interpretation of <span class="math">\(\delta\)</span> captures this nicely. Since we inverted the
curve, if <span class="math">\(\mathcal{L}=\infty\)</span>, we simply have <span class="math">\(e^\varepsilon/e^\mathcal{L}=0\)</span>.
The distance to <span class="math">\(1\)</span> is exactly <span class="math">\(1\)</span>, so we count these events with maximal
weight. The graph looks like this:</p>
<p><center>
<img alt="Plotting exp(ε)/exp(PLRV) and highlighting the area under 1 when that function is 0 below 0.006 and 1 everywhere else" src="https://desfontain.es/privacy/images/exp-plrv-distinguishing.svg">
</center> </p>
<p>In that case, <span class="math">\(\delta_1=\delta_2\)</span>: all "bad" events are worst-case events. For
such a mechanism, the two characterizations of <span class="math">\(\delta\)</span> are the same.</p>
<h1 id="final-note">Final note</h1>
<p>You might be wondering: why use Gaussian noise at all if it requires <span class="math">\(\delta>0\)</span>?</p>
<p>This is an excellent question. I'm glad you asked it, because it is exactly the
topic of the <a href="gaussian-noise.html">next blog post</a> in this series. Or you can, as always,
select another article to read next in the <a href="friendly-intro-to-differential-privacy.html">table of contents</a>!</p>
<hr>
<p><small>
Thanks to <a href="http://www0.cs.ucl.ac.uk/staff/s.meiser/">Sebastian Meiser</a>, who
wrote the <a href="https://eprint.iacr.org/2018/277.pdf">reference paper</a> about the
subtleties with <span class="math">\(\delta\)</span>. It makes for excellent reading if you want to dig a
bit deeper into this. Thanks also to Antoine Amarilli for proofreading this
blog post, and to Ivan Habernal for detecting a mistake in an earlier version.
</small></p>
<script type="text/javascript">
var button = document.getElementById('toggleProof');
var defaultButton = 'Show me the proof';
button.innerHTML = defaultButton
button.addEventListener('click', function (event) {
button.innerHTML = button.innerHTML == defaultButton ? 'Hide the proof' : defaultButton;
proof = document.getElementById('proof');
proof.style.display = proof.style.display == 'none' ? 'block' : 'none';
});
</script>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>A reading list on differential privacy2019-09-20T00:00:00+02:002019-09-25T00:00:00+02:00Damien Desfontainestag:desfontain.es,2019-09-20:/privacy/differential-privacy-reading-list.html<p>A list of papers to check out to learn more about differential privacy.</p><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p></small></p>
<hr>
<p><span class='lettrine'>S</span><strong>omeone</strong> recently asked me for reading
suggestions to learn more about differential privacy (DP). I thought that the
answer was worth posting somewhere. So, here it is: a list of material that I
often recommend to people who want to dive into the field.</p>
<p>Before I start, two remarks. First, most of these are going to be research
papers. There's a lack of accessible material on differential privacy. I hope it
gets better over time; I'll keep this list updated as I discover more good
stuff. Second, this list is going to be <em>very</em> subjective. I'm more interested
in some problems than others. I like beautiful theory and very practical
applications. I'm less enthusiastic about the research in-between. This list
might not be the right one for you. If you're a student starting a research
project, ask your advisor what you should read! They'll be a better source of
reading material than a random guy on the Internet.</p>
<h1 id="introductory-and-reference-material">Introductory and reference material</h1>
<p>There are less links that I'd like in the category « suitable for a
non-technical audience ». That's exactly why I'm writing these blog posts…
Still, some people have created great material.</p>
<p>I recently stumbled upon a fantastic 12-minute video called <a href="https://www.youtube.com/watch?v=pT19VwBAqKA">Protecting Privacy
with MATH</a>. It's done by <a href="https://en.wikipedia.org/wiki/MinutePhysics">MinutePhysics</a>, and it's friendly,
understandable, yet very accurate. I love that it explains the basics of
reconstruction attacks! It provides a great motivation for differential privacy.</p>
<p>For more in-depth material, I recommend the <a href="https://scholarship.law.vanderbilt.edu/jetlaw/vol21/iss1/4/">Primer for a Non-Technical
Audience</a>. It's written by folks at the Harvard <a href="https://privacytools.seas.harvard.edu/">Privacy Tools
Project</a>, and is perfect if you don't want heavy math. I prefer it to
the typical recommendation for a reference book on DP, surnamed the « Privacy
Book ». Its real name is <a href="https://www.cis.upenn.edu/~aaroth/privacybook.html">The Algorithmic Foundations of Differential
Privacy</a>. One of the authors is Cynthia Dwork, one of the creators
of DP, so it's a good way of discovering the original motivations… But I would
mostly recommend Sections 1 to 3, not the whole thing.</p>
<p>Ah, on the topic of things written by the creators of DP! <a href="https://github.com/frankmcsherry/blog">Frank McSherry's
blog</a> makes for a fun read (look for the articles tagged « Privacy »).</p>
<h1 id="local-and-shuffled-model">Local and shuffled model</h1>
<p>Excited by my [blog post][local] about the different models of DP? When it comes
to the privacy model, the <a href="https://arxiv.org/abs/1407.6981">RAPPOR</a> paper is a classic. It explains how Google
Chrome collected useful statistics without learning information about
individuals. The follow-up paper, <a href="https://arxiv.org/abs/1710.00901">Prochlo</a>, introduced the shuffled model in a
very practical context. To understand its theoretical guarantees, I recommend
the later <a href="https://arxiv.org/abs/1903.02837">Privacy Blanket</a> paper, it provides a very nice explanation
of the intuition behind it<sup id="fnref:shuffling"><a class="footnote-ref" href="#fn:shuffling">1</a></sup>.</p>
<h1 id="rolling-out-differential-privacy">Rolling out differential privacy</h1>
<p>Moving on to applied views of central DP for statistics. Google colleagues and I
recently published recently a paper about a <a href="https://arxiv.org/abs/1909.01917">differentially private SQL
engine</a>. This isn't the first work in that direction<sup id="fnref:sql"><a class="footnote-ref" href="#fn:sql">2</a></sup>, but I'm still
pretty proud of it: we worked hard to make it easy to understand and mostly
self-contained! You can see an updated version in Section 4.2 of my <a href="https://desfontain.es/thesis/">PhD
thesis</a>. Further in the thesis, you can also find a discussion of
<a href="https://desfontain.es/thesis/OperationalAspectsOfAnonymization.html">operational aspects of DP</a>, a topic that doesn't get as much
attention as it probably should.</p>
<p>Want more about rolling out DP in practice? The <a href="https://arxiv.org/abs/1809.02201">Issues Encountered Deploying
Differential Privacy</a> paper, by the US Census Bureau, is a must-read. I
strongly relate to the problems it raises. For even more fun and absurd
implementation things, go read about the <a href="https://www.microsoft.com/en-us/research/publication/on-significance-of-the-least-significant-bits-for-differential-privacy/">floating-point attack</a> on
the Laplace mechanism, it's a classic. </p>
<h1 id="theory-of-differential-privacy">Theory of differential privacy</h1>
<p>There are natural theory questions raised by DP. For example « how many queries
can we answer privately and accurately on the same database? » <a href="https://privacytools.seas.harvard.edu/publications/complexity-differential-privacy">The Complexity
of Differential Privacy</a> is a survey of fundamental results on
accuracy/privacy trade-offs. This being theory, almost everything in there is
asymptotic, of course.</p>
<p>The theory around the <em>privacy loss distribution</em> is beautiful. It gives a
global view of the privacy leakage of a mechanism. It's less simple, but it
makes much more sense than only considering a pair of (ε,δ) parameters. To learn
about it, I recommend the <a href="https://eprint.iacr.org/2018/820">Privacy Loss Classes</a> paper. It has practical
consequences, too, on estimating privacy budgets over many compositions. It
enables neat ways of estimating the evolution of a privacy budget over many
compositions. This is explored in the <a href="https://eprint.iacr.org/2017/1034">Privacy Buckets</a> paper, which is
also worth a read.</p>
<p>Speaking of interesting reformulations of DP, the hypothesis testing formulation
is also super cool. The <a href="https://arxiv.org/abs/1905.02383">Gaussian Differential Privacy</a> paper does a
great job at explaining it. It also gives you nice results on <em>amplification by
sampling</em>, a shiny and useful tool in practice<sup id="fnref:sampling"><a class="footnote-ref" href="#fn:sampling">3</a></sup>.</p>
<p>Finally, there are interesting debates around how to interpret the guarantee
provided by DP. Everyone understands the math, but how do we translate it in
practice? There's a long-standing debate about data correlations and adversary
strength. On that topic, the <a href="https://arxiv.org/abs/1710.05899">Differential Privacy as a Causal Property</a>
paper is fantastic. It clarifies this debate, links to all relevant references,
and proposes a neat way to solve this issue conceptually.</p>
<h1 id="differentially-private-machine-learning">Differentially private machine learning</h1>
<p>I don't know much about machine learning, but it felt weird to not mention it at
all in an article like this. So I asked my colleague <a href="https://ai.google/research/people/PeterKairouz">Peter Kairouz</a> to
suggest some material! The following suggestions are from him. I was glad to see
that there aren't only scientific papers in there!</p>
<p>First, start with this pretty great 2-hour <a href="https://www.ece.rutgers.edu/~asarwate/nips2017/">video tutorial</a> (with
available slides and historical references). If you then want to actually test
this in practice, try out <a href="https://github.com/tensorflow/privacy">TensorFlow Privacy</a>! For a nice introduction to
it, check out this <a href="http://www.cleverhans.io/privacy/2019/03/26/machine-learning-with-differential-privacy-in-tensorflow.html">blog post</a>, by one of the TensorFlow Privacy
authors. It explains how to run DP-SGD with it, step by step and with a lot of
additional references.</p>
<p>The paper that introduced DP-SGD is a classic: <a href="https://arxiv.org/abs/1607.00133">Deep Learning with Differential
Privacy</a>. Another method for the same problem, DSSGD, appeared earlier in
<a href="https://www.comp.nus.edu.sg/~reza/files/Shokri-CCS2015.pdf">Privacy-Preserving Deep Learning</a><sup>(PDF)</sup>. Yet another
alternative approach, more generic and surprisingly understandable, is called
PATE. It's described in a <a href="http://www.cleverhans.io/privacy/2018/04/29/privacy-and-machine-learning.html">blog post</a>, which comes with references to the
original papers.</p>
<p>Most of these methods rely on tight accounting on the privacy budget. A common
tool for that is <a href="https://arxiv.org/abs/1702.07476">Rényi Differential Privacy</a>, a DP variant that
considers the <em>averaged</em> privacy loss. Another is <a href="https://arxiv.org/abs/1808.06651">Amplification by
Iteration</a>, which studies mechanisms that iterate over the data
multiple times, without releasing intermediary results. Both techniques enable
better composition results.</p>
<p>Deep learning isn't the only interesting ML application of differential privacy.
Methods also exist for <a href="https://arxiv.org/abs/1507.02482">Ordinary Least Squares</a>, or for <a href="https://arxiv.org/abs/0912.0071">Empirical Risk
Minimization</a>. This last method admits <a href="https://arxiv.org/abs/1405.7085">Efficient Algorithms and Tight
Error Bounds</a>. In general, <a href="https://arxiv.org/abs/1908.09970">Convex Optimization</a> problems are a nice
fit for DP methods.</p>
<h1 id="something-missing">Something missing?</h1>
<p>If you feel this reading list is missing something, please let me know! I'd like
this post to be a living resource. So I'm enthusiastic about adding new
material, especially if they're more approchable than scientific papers. My
contact info is in the footer of this page.</p>
<hr>
<p><small>Thanks to Antoine Amarilli, Úlfar Erlingsson, Peter Kairouz, and Rob
Yoder for their helpful comments and suggestions.</small></p>
<div class="footnote">
<hr>
<ol>
<li id="fn:shuffling">
<p>Amplification by shuffling was initially formalized in the <a href="https://arxiv.org/abs/1811.12469">eponymous
paper</a>. <a class="footnote-backref" href="#fnref:shuffling" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:sql">
<p>The first system to do generic differentially private computations was
<a href="https://www.microsoft.com/en-us/research/project/privacy-integrated-queries-pinq/">PINQ</a>, and a major follow-up was <a href="https://arxiv.org/abs/1706.09479">Flex</a>. Both papers are worth a read too! <a class="footnote-backref" href="#fnref:sql" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:sampling">
<p>For a slightly more exhaustive and historical view of results in
that area, Section 4 of the <a href="https://arxiv.org/abs/1807.01647">Amplification by Subsampling</a> paper
(and the corresponding references) is a solid resource. <a class="footnote-backref" href="#fnref:sampling" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>« What does a privacy engineer do, anyway? »2019-08-13T00:00:00+02:002018-08-24T00:00:00+02:00Damien Desfontainestag:desfontain.es,2019-08-13:/privacy/privacy-engineer.html<p>Besides having a super cool job title, what is it like being a privacy engineer?</p><p><strong>This</strong> is another one of <a href="part-time-phd.html">those posts</a> where I answer a
question I get regularly. What is it like being a privacy engineer? Besides a
cool job title, what does the job actually entail?</p>
<p>Before I even start, I need to point out a few major caveats of this post.</p>
<ul>
<li>My experience is not representative of privacy engineering as a whole. I'm
sure it's different across companies, and even across people within the same
company.</li>
<li>I have a particularly narrow view. I work on a sub-field of privacy
(anonymization), and I've never led a team or hired privacy engineers.</li>
<li>"Privacy engineer" is not even my official job title. I still have my initial
job title at Google: software engineer (<a href="part-time-phd.html">part-time</a>).
Privacy engineering is definitely the job I do, though — they don't care
much about job titles. I might switch at some point.</li>
<li>Like in all other entries from this blog, all opinions are mine and mine only.
I'm only talking about my own experiences and feelings, and this post isn't
vetted by my employers.</li>
</ul>
<p>So, what do I do, on a daily basis, while working as a privacy engineer? I would
split the job responsibilities in three broad categories.</p>
<h2 id="consulting-with-teams">Consulting with teams</h2>
<p>A large part of my time is spent helping product teams get privacy right.</p>
<p>Back when I did generic privacy reviews, I checked future product launches for a
wide variety of things. Will users understand what happens with the data they
share? Is the data appropriately protected in transit and on-rest? Can the
product be misused and allow bad people to do evil<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>? Is the system collecting
only what is required for the product's functionality? Will it have a harmful
behavior in specific cases, or will it protect at-risk users properly<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup>? Are
deletions handled correctly? </p>
<p>Some of this is compliance: making sure that the product aligns with existing
policies and regulations is important. This often isn't the focus point, though.
Everything that can be ticked off a checklist is usually pretty straightforward.
What's most complex and interesting is to identify what can go wrong in specific
scenarios.</p>
<p>How do I do this in practice? I read through design docs, slide decks and
sometimes code or demos to understand the product. I ask for additional
documentation when it's lacking. And poke at the system to see where it could
fail in unexpected and problematic ways. Then, I'll communicate the findings to
the team, and help them correct possible issues.</p>
<p>Nowadays, I'm in a group focused on anonymization, so I have a narrower focus.
Product teams come to us when they need to anonymize data, and we help them get
it right. We make sure they understand what they need to do and how to do it.
Then, we give them the green light when we end up on a solution we're
comfortable with.</p>
<p>This process is much easier when teams consult with privacy folks as soon as
possible. I love to be involved in early design discussions! It's in everyone's
interest. It avoids making choices that we'll regret later, which can save a lot
of engineering time. And if the team does the right thing in the first place,
that makes my job much easier at review time!</p>
<p>This part of the job requires lots of empathy. It's necessary to relate with
users, and understand what will create issues. It's also crucial to build
productive and respectful relationships with product teams. Luckily for me, it's
something that can be learned and improved over time. I wasn't very good at it
at first!</p>
<h2 id="building-things">Building things</h2>
<p>This is why the job title has « engineer » in it!</p>
<p>Finding issues in products is only the first step. Can we automate some of these
investigation methods? Make sure certain classes of problems don't happen in the
future? Detect failures early to prevent them from causing harm? These follow-up
questions can lead to impactful engineering projects.</p>
<ul>
<li><em>Technical improvements to processes</em>: this is not a phenomenon limited to
privacy. When engineers run processes, they'll detect inefficiencies, and
identify automation opportunities. Building tooling to assist with
checklist-type things is often a good idea. Time is better spent focusing on
the complex and unique aspects of consultations!</li>
<li><em>Improving infrastructure</em>: baking privacy into your tech stack is an
excellent investment. Suppose that some vetted storage system takes care of
encryption and deletion correctly. You no longer need to worry about these
aspects in a product that uses this system. That's a great way to save time
and avoid problems! Privacy engineers are uniquely positioned to notice when
this is worth doing.</li>
<li><em>Monitoring</em>: how do you check that products continue to behave as expected
over time? Some problems might arise after the privacy review. Bugs happen.
Code evolves over time. Changes might appear harmless, but have unintended
consequences. Catching potential failures with automated monitoring before
they harm anyone is very rewarding. And again, to know what to look for and
how to detect privacy issues, you need domain experts.</li>
</ul>
<p>Sometimes, it makes sense for a privacy engineer to take on such projects on
their own. This is especially true when there is a lot of specific expertise
involved, which is the case for my group: a large chunk of our work is about
building tools to make anonymization easier and safer. Sometimes, building a new
thing yourself is not the optimal move. It might make more sense to collaborate
with existing teams, or influence their roadmap.</p>
<p>This part of the job is like a classic software engineering job: writing design
docs, getting resources, writing and reviewing code, improving documentation,
writing tests… In my experience, though, there's often more communication
involved than on a typical project. The problems tackled are often very
horizontal: pain points shared across an entire organization, a new rule that
applies across the board… In these cases, maintaining technical alignment
between related efforts is crucial. Privacy engineers are in the perfect central
position to help with this aspect.</p>
<h2 id="setting-policy">Setting policy</h2>
<p>Privacy engineers try to make their organization do the right thing. But
who decides what the « right thing » is in the first place?</p>
<p>Turns out, it's also part of the job. Privacy engineers take high-level goals or
regulations and translate them into concrete, actionable requirements. Typical
privacy principles are broad and vague in nature, and have many interpretations.
This isn't great: people will have some <em>creative</em> ideas about how to put them
in application.</p>
<p>Being at the interface between non-technical stakeholders and engineers is
tricky, but necessary: if nobody does this job, it's not going to end well. And
it goes both ways! Helping policy makers understand what makes sense from a
technical perspective is crucial. Otherwise, you end up with inapplicable rules,
or counter-productive efforts.</p>
<p>In addition to that, policies don't all come from top-down regulations or
principles. It's frequent to stumble upon questions for which there is no
existing guidance. You often have to make judgment calls, and when you do, it's
important to document these decisions: this is the only way to keep them
consistent across products and over time. And doing that is, in effect, setting
an unofficial policy. Depending on how generalizable it is, it might be worth
turning it into official guidance.</p>
<p>I find policy work <em>way</em> more complex than consulting or engineering work. It
involves long discussions with a wide variety of people: lawyers, executives,
engineers, product managers… And it's critical to get right. Spending the time
to write great guidance is an investment that pays back many times over. And
inconsistent or inapplicable policies can have a huge damaging impact<sup id="fnref:3"><a class="footnote-ref" href="#fn:3">3</a></sup>. So
it's sometimes frustrating, but also very challenging and rewarding.</p>
<p>In practice, what does policy work look like? Many meetings, long discussions on
docs, unending email threads. Yum!</p>
<h2 id="all-the-other-stuff">All the other stuff</h2>
<p>I lied! There's a fourth category. It's a catch-all, for all the extra
responsibilities besides the core ones.</p>
<ul>
<li><em>Education</em>: you can't make every person in your organization a privacy
expert, but it doesn't hurt to try! Giving talks and doing outreach is useful
for many reasons. It helps orient people in the right general direction when
designing new products. It increases their awareness of potential issues, and
makes them more likely to consult with experts early. It's also a great way to
recruit! People you reach this way might later join your team, or become local
privacy experts in their own team.</li>
<li><em>Proactive investigations</em>: poking at existing products outside of structured
consultations can be worthwhile. Especially if nobody has looked at them in a
while…</li>
<li><em>User advocacy</em>: privacy engineers try to make sure products aren't harmful to
marginalized communities. As such, they have a duty to speak up when that's
the case, even if it's not technically about privacy<sup id="fnref:4"><a class="footnote-ref" href="#fn:4">4</a></sup>.</li>
<li><em>Incident response</em>: bad stuff happens, and you need to have a process to make
sure some people are ready to put out fires! Privacy engineers can be in such
incident response roles. I never had such a role, though, so I don't know what
it's like.</li>
<li><em>External outreach</em>: few privacy engineers do external outreach, even though
they're in a unique position to do so. We could do better in that area, and
collaborate more with academia and civil society. Luckily, most excellent
folks are actively <a href="https://www.usenix.org/conference/pepr19">improving</a> the
situation on that front!</li>
<li><em>And a bunch of other engineering-related things</em>. You sometimes need to
analyze data to quantify and prioritize issues. You want to keep educating
yourself on new developments in the privacy space. You also need to pay
attention to public discourse; it's crucial to better understand users, and
anticipate new regulations. You sometimes want to run user studies to align
your UIs with user expectations.</li>
</ul>
<p>I'm sure that I'm forgetting some aspects. You might be a privacy engineer and
this list might not feel very familiar. If that's the case, drop me a line! I'd
love to understand other perspectives on that topic. I hope you agree with me on
one thing, though: being a privacy engineer is challenging, fun and rewarding!</p>
<p>Now, there's a related question I also get sometimes: « being a privacy engineer
sounds awesome, how can I become one ? ». Unfortunately, I don't have a great
answer to that one. For me, it was pretty random. I joined Google as a software
engineer, switched to the privacy team because it sounded fun, and learned
everything I know there. I've heard good things about a <a href="http://privacy.cs.cmu.edu/">Master's degree at
CMU</a>… But apart from that, I don't know a lot of
educational opportunities. I'm not sure how to hunt for privacy engineering
jobs, either. I hope someone else writes a good answer to that question! Because
we definitely need more privacy engineers in industry.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>There are lots of sub-questions here! Can a domestic abuser use the
product to spy on their spouse? Can criminals take advantage of it to run
online scams? Can political actors use it to spread disinformation? <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>Will your system deadname trans people, or unmask political activists? Are
privacy-critical surfaces accessible and understandable to people with
disabilities? Can the product amplify hurtful or triggering content? Needless
to say, this is why you need your privacy engineering team to be diverse. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>Dealing with
<a href="https://en.wikipedia.org/wiki/Technical_debt">technical debt</a> in software
engineering is no easy feat… But "policy debt" is many times worse and more
difficult to pay back. Significantly changing existing policies is <em>hard</em>. <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:4">
<p>Things like ethics or ML fairness deserve specific processes and technical
expertise. They're like security and privacy, complex and crucial to get
right. But when these processes don't exist, they fall by default under the
privacy umbrella. So existing practitioners have a duty to keep an eye on this
sort of thing in the meantime. <a class="footnote-backref" href="#fnref:4" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
</ol>
</div>Local vs. central differential privacy2019-06-27T00:00:00+02:002021-09-30T00:00:00+02:00Damien Desfontainestag:desfontain.es,2019-06-27:/privacy/local-global-differential-privacy.html<p>Differential privacy is used in two very distinct contexts. Come learn about the distinction between the two, and interesting new directions that combine them!</p><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p></small></p>
<hr>
<p><span class='lettrine'>W</span><strong>hen</strong> people talk about differential privacy,
they don't always have the same thing in mind. People agree on the
<a href="differential-privacy-in-more-detail.html">definition</a>, but context also matters a <em>lot</em>. There are two "flavors"
of differential privacy: the <strong>central model</strong> (or <strong>global model</strong>) and the
<strong>local model</strong>. In this post, I'll first explain the difference between the two
options. Then, I'll give a glimpse of an exciting new idea that combines them.</p>
<p>What do I mean by "context"? In abstract terms, a differentially private
algorithm transforms an input into an output.</p>
<p><center>
<img alt="Diagram "raw input" → "anonymized output", with "magic" on the arrow." src="https://desfontain.es/privacy/images/input-output-diagram.svg">
</center></p>
<p>The main difference will be: who has access to the raw input?</p>
<h1 id="central-differential-privacy">Central differential privacy</h1>
<p><a name="central"></a></p>
<p>In the <em>central model</em> (or <em>global model</em>) of differential privacy, a <em>central
aggregator</em> has access to the real data. What is this "aggregator"? Typically,
it's a service or a research organization collecting data about individuals. In
this model, each user sends their data to this aggregator without noise. The
aggregator takes this data, and transforms it with a differentially private
mechanism.</p>
<p><center>
<img alt="Diagram "users" → "aggregator" → "anonymized output", with "magic" on the second arrow." src="https://desfontain.es/privacy/images/global-dp-diagram.svg">
</center></p>
<p>The differentially private mechanism is only applied <em>once</em>, at the end of the
process. The aggregator can then e.g. publish the result or share it with
third parties.</p>
<p>This model has one big advantage: <strong>accuracy</strong>. In the central model, you usually
don't need to add a lot of noise to get valuable results with a low
<span class="math">\(\varepsilon\)</span>. Remember when I explained how to release <a href="differential-privacy-in-practice.html">statistics</a>
or <a href="almost-differential-privacy.html">histograms</a> using differential privacy? These examples were using
this central model. It worked pretty well: only a <a href="differential-privacy-in-practice.html#laplace">little noise</a> was
needed to hide someone in a count.</p>
<p>Where's the catch, then? Well, the central aggregator needs to know the real
data. In the scenarios above, we added noise to <em>real counts</em>. This is only
possible if we know the true numbers in the first place… To enable this, <strong>each
user has to trust the aggregator</strong> enough to share data with it. That might be
difficult: the aggregator can be an untrusted company or government. Also, with
the central model, all the data is collected in one place. It increases the risk
of catastrophic failure, for example if the aggregator gets hacked and leaks all
the data.</p>
<p>The most famous real-world example of the central model is probably the US
Census. In 2020, the US Census will use differential privacy to anonymize the
data before publication. This is pretty exciting! You can read more about it
<a href="https://www.census.gov/newsroom/blogs/random-samplings/2019/02/census_bureau_adopts.html">here</a>.</p>
<h1 id="local-differential-privacy">Local differential privacy</h1>
<p><a name="local"></a></p>
<p>What's the alternative, then? It's the <em>local model</em> of differential privacy. In
this model, there is still an aggregator, but they no longer have access to the
real data. Instead, each user applies a differentially private mechanism to
their <em>own</em> data. And they only send their data to the aggregator once it's
already anonymized.</p>
<p><center>
<img alt="Diagram "users" → "aggregator" → "anonymized output", with "magic" on the first arrow." src="https://desfontain.es/privacy/images/local-dp-diagram.svg">
</center></p>
<p>After collecting this noisy data, the aggregator can compute some statistics,
and publish them. This last step doesn't need to be differentially private: the
data is anonymous to begin with. In theory, the aggregator could publish the
entire dataset they collected.</p>
<p>The big advantage of this model is that it <strong>no longer requires trust</strong>. Since
each user is protecting their own data, they're safe even if the aggregator is
malicious. This makes the local model well-suited to situations where trust is
difficult to get. And we already saw an example of this! Remember the survey
about drug use that used <a href="differential-privacy-in-more-detail.html#rr">randomized response</a> to gather data. The scheme
allowed subjects to answer honestly without admitting to breaking the law. This
is a typical application of the local model.</p>
<p>Can you guess the drawback of this model? Since each user must add noise to
their own data, <strong>the total noise is much larger</strong>. You typically need many more
users than in the central model to get useful results. To mitigate this problem,
practical applications often use high values of <span class="math">\(\varepsilon\)</span>.</p>
<p>Besides randomized response, the most famous example of this model is probably
<a href="https://github.com/google/rappor">RAPPOR</a>. This clever scheme was invented to collect differentially private data
in Google Chrome. Another example, a bit more recent, is the <a href="https://machinelearning.apple.com/2017/12/06/learning-with-privacy-at-scale.html">mechanism</a>
that Apple uses to collect data on the iOS keyboard.</p>
<h1 id="the-best-of-both-worlds">The best of both worlds</h1>
<p><a name="shuffling"></a></p>
<p>Until very recently, there was no middle ground between the two options. The
choice was binary: either accept a much larger level of noise, or collect raw
data. This is starting to change, thanks to recent work on a novel type of
architecture. This new design is called <em>ESA</em>: Encode, Shuffle, Analyze. In the
two previous models, there was only two types of participants: users and
aggregator. By contrast, the ESA architecture has three elements.</p>
<ul>
<li>The <em>encoder</em> is a fancy name to say "user". It collects the data, encrypts it
twice, and passes it to the shuffler.</li>
<li>The <em>shuffler</em> is an intermediary process. First, it removes identifiers, and
groups similar pieces of data together. Then, it passes these pieces of data
to the analyzer if there are enough of them.</li>
<li>The <em>analyzer</em> actually decrypts the data, and computes the statistics we're
interested in.</li>
</ul>
<p><center>
<img alt="Diagram "users" → "shuffler" → "analyzer" → "anonymized output", with "magic" on the shuffler and analyzer boxes." src="https://desfontain.es/privacy/images/esa-diagram.svg">
</center></p>
<p>In ESA, the data is encrypted twice, in two layers. The shuffler can only
decrypt the first layer. It contains the user IDs, and something called "group
ID". This group ID describes what <em>kind</em> of data this is, but not what is the
actual <em>value</em> of the data. For example, group ID could be a label like "app
latency for iOS", while the actual data would be the latency value.</p>
<p>The shuffler then groups all group IDs together and counts how many users are in
each group. If there are enough users in a group, it passes them all to the
analyzer. The analyzer can then decrypt the second layer of the data, and
compute the output.</p>
<p>Having two layers allows to separate the data and its metadata. The shuffler can
see IDs, but not the actual data. Meanwhile, the analyzer only sees the data in
batches, and cannot know who sent what. The magic comes with tweaking what the
shuffler does. You add some randomness in the process, in well-chosen places…
And it ensures that the final output is differentially private.</p>
<p>This nice property comes from the separation between shuffler and analyzer. Of
course, if the same entity runs both, that benefit disappears. This design can
still be useful! In particular, catastrophic failures are much less likely. But
from a trust perspective, it becomes like the central model. So there are two
options:</p>
<ul>
<li>A different organization can run the shuffler as a service. In principle, a
nonprofit like the <a href="https://www.eff.org/">EFF</a> could play this role.</li>
<li>The shuffler can run on secure hardware, using e.g. <a href="https://en.wikipedia.org/wiki/Trusted_Computing#Remote_attestation">remote attestation</a>. This
way, the client can verify that the shuffler does what it's supposed to.</li>
</ul>
<p>Both options are probably a few years away, but I find them pretty exciting.
They could, in theory, bring the best of both worlds: the low noise levels of
the central model, and the trustless aspect of the local model. If you want to
learn more about it, you can check out the <a href="https://arxiv.org/pdf/1710.00901.pdf">paper</a> that introduced this design.</p>
<hr>
<p>Interested in learning more about differential privacy? Head over to the <a href="friendly-intro-to-differential-privacy.html">table
of contents</a> of this series to see its other posts. Or you can directly
go to the <a href="why-not-differential-privacy.html">next article in the series</a>, which is somewhat paradoxical:
it explores what it means for an algorithm to <em>not</em> be differentially private.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Research post: Cardinality Estimators do not Preserve Privacy2019-03-09T00:00:00+01:002019-08-24T00:00:00+02:00Damien Desfontainestag:desfontain.es,2019-03-09:/privacy/cardinality-estimators.html<p>You can't both remember unique individuals and not remember them. Shocking, right? :D</p><p><strong>Next</strong> summer at <a href="https://petsymposium.org/index.php">PETS</a> 2019, I'll present a <a href="https://arxiv.org/abs/1808.05879">paper</a> I wrote with <a href="http://www.andreas-lochbihler.de/">Andreas
Lochbihler</a> and my PhD advisor <a href="https://www.inf.ethz.ch/personal/basin/">David Basin</a>. This post will
attempt to explain what the paper is about, and what its results mean in
practice.</p>
<h1 id="tldr">tl;dr</h1>
<p>You can't both remember unique individuals and not remember them. </p>
<p>It's not incredibly surprising, but still cool to have a formal negative result
about it :D</p>
<h1 id="cardinality-estimators">Cardinality estimators</h1>
<p>The title of the paper is <em>Cardinality Estimators do not Preserve Privacy</em>.
First, what's a cardinality estimator? It's an algorithm, and an associated data
structure called a <em>sketch</em>. It does two things:</p>
<ul>
<li>it can <em>count unique items</em> in a list (without duplicates);</li>
<li>and you can <em>merge</em> several of them to count unique items in multiple lists.</li>
</ul>
<p>Suppose you have two huge log files, each containing one billion unique
identifiers. How many unique identifiers are present in the union of both files?
If you only have counts, there's no way to tell. Both files might have the same
identifiers, so the total count is one billion. Or maybe they only contain
different identifiers, so the total count is two billion. Or it could be
in-between these two extreme options.</p>
<p>A cardinality estimator can answer that question. You can apply it to each log
and get two sketches. Each can estimate the number of unique items in its input
log. Further, you can merge the two sketches to get a new sketch… And the
estimated count of this new sketch is the <em>deduplicated</em> version of the two
counts.</p>
<p>The simplest example is a <a href="https://en.wikipedia.org/wiki/Set_(abstract_data_type)">set</a> that remembers all the items it saw. It will
return an exact count, and merging two sketches is straightforward.
Unfortunately, it doesn't scale very well: you need a lot of memory to remember
every single element, and lots of time to merge large sketches. Instead, we
often trade precision for performance, and use <em>approximate</em> algorithms. They
don't return an exact answer, but are much faster and memory-efficient.</p>
<p>The first cardinality estimator I encountered is <em>HyperLogLog</em>. It's a very
popular choice: very efficient, and precise enough for most use cases. The idea
behind it is quite smart: if you want to learn about it, I recommend this <a href="https://research.neustar.biz/2012/10/25/sketch-of-the-day-hyperloglog-cornerstone-of-a-big-data-infrastructure/">blog
post</a> and this <a href="https://ai.google/research/pubs/pub40671">research paper</a>.</p>
<p>As part of my privacy reviewer role at Google, a team approached me with a
question. They were storing HyperLogLog sketches, and asked me: "are those
sketches as sensitive as the original data?".</p>
<h1 id="privacy-modeling">Privacy modeling</h1>
<p>To answer that question, one first has to define what we mean by sensitive. We
used the idea behind differential privacy, the attacker's <a href="differential-privacy-in-more-detail.html">information
gain</a>. Intuitively, if an attacker has some suspicion that a user was
added to a sketch, and then looks at the sketch… How much can their suspicion
increase, or decrease?</p>
<p>However, we added two twists to this idea.</p>
<ul>
<li>First, we only care if the level of suspicion <em>increases</em>, not if it
decreases. If the attacker learns that their target is in a given sketch, it's
a problem. If they learn that their target is <em>not</em> in a given sketch, then we
don't consider it that big of a deal.</li>
<li>Second, we don't assume that the attacker knows all users except their target.
Instead, we assume that the attacker knows <em>nothing</em> about the sketch. From
their perspective, the data is 100% random.</li>
</ul>
<p>These choices only made sense for the original use case (the Google privacy
review). For the sketches I was considering, learning that somebody was <em>not</em> in
a sketch wouldn't have been very problematic. There was also no reason to assume
the attacker would have any background knowledge. These assumptions are much
weaker than in most data privacy papers and use cases.</p>
<p>If this research had led to a positive result, it wouldn't have been very
convincing. "Here is a cardinality estimator that satisfies this super weak
notion of privacy!" People would have pushed back, saying that the assumptions
were too weak.</p>
<p>Luckily (for me), the result was otherwise. First, I found that HyperLogLog was
not private according to this definition. That was the easy part, and it led to
a natural follow-up question: can we make it private? Or more generally, can we
build a private cardinality estimator? We want it to have the same nice
properties as HyperLogLog, but a better privacy. </p>
<h1 id="main-result">Main result</h1>
<p>It turns out that the answer is negative. Even with our weak privacy notion, the
problem is <em>unsolvable</em>. No cardinality estimator can be both private and
accurate. A private cardinality estimator has its accuracy gets exponentially
worse with its size.</p>
<p>Since the result is negative, the privacy definition's weakness makes our result
<em>stronger</em>. Accuracy is incompatible with a weak notion of privacy… So it's also
incompatible with stronger notions. We also considered even weaker variants,
e.g. allowing for a small <a href="almost-differential-privacy.html">probability of error</a>. It didn't change the
negative result. There seems to be a fundamental incompatibility between privacy
and accuracy.</p>
<p>There is one caveat: this is only true if you want to be able to merge an
arbitrary number of sketches. If the accuracy can get worse as you merge
sketches, the result does not hold. In such a context, privacy-preserving
schemes might exist. So, if your use case only requires you to merge a bounded
number of sketches, you might have options. But if you want analysts to be able
to do arbitrary aggregations of sketches and still get reasonably good results…
then privacy is an impossible goal.</p>
<p>With this added caveat, our result becomes more intuitive. To merge two sketches
that count unique users, you have to deduplicate users… So you have to keep the
information about which users are in the sketch. As with HyperLogLog, this
information doesn't have to be exact. But the more you remember, the more an
attacker can use it to break the privacy property. HyperLogLog remembers some
users more than others, and that's what allows it to stay accurate. If you can't
remember any user well, then your cardinality estimator gets very inaccurate.</p>
<p>So there are two contributions: a theoretical one and a practical one. </p>
<p>The theoretical part is a confirmation and formalization of an expected
phenomenon. It's still interesting, because it's quite rare. There aren't many
negative results in the world of differential privacy. A typical privacy paper
takes a problem and solves it in a differentially private way. Here, we're
presenting a problem for which this is impossible. This leads to an open
question, which we ask at the end of the paper: what's a minimal set of
constraints that make differential privacy impossible?</p>
<p>There is also one practical consequence: cardinality estimators in use today are
not private. Their sketches should be considered roughly as sensitive as raw
data. We proved it manually for HyperLogLog… But our result is generic, so it
holds for all cardinality estimators.</p>
<h1 id="behind-the-scenes">Behind the scenes</h1>
<p>The story we tell in the paper isn't exactly the path we actually followed to
get our results. For example, the attacker's lack of background knowledge came
from a practical constraint. In the original problem, cardinality estimators
were built using large-scale tools like <a href="https://en.wikipedia.org/wiki/MapReduce">MapReduce</a>. Such tools assume that the
aggregation primitives are <em>deterministic</em>: for example, MapReduce double-checks
the computation results for fault tolerance. HyperLogLog is deterministic: a
sketch formed with a given input is always the same. All other cardinality
estimators we found were also deterministic. So at first, we required that any
solution to our question should also be deterministic.</p>
<p>But it's impossible to get differential privacy without adding noise to the
data. We assumed that the attacker lacked knowledge about the data to get around
this problem. If the data itself is random, it can play the same role as noise
from the attacker's perspective.</p>
<p>The first negative result we got had the assumption that cardinality estimators
were deterministic. I was unhappy about it, and wanted the paper to have a more
generic result. For a good chunk of 2017, I tried to extend the result to
arbitrary cardinality estimators. We gave up and tried to submit the paper in
autumn, but got rejected for exactly this reason. This gave me a motivation
boost: reviewers, too, thought this was important. The next submission had the
generalized result. A good example of the peer review process working well! ^^</p>
<p><em>August 2019 edit: this paper and the presentation I gave to PETS won a
<a href="https://petsymposium.org/student-paper-award.php">Best Student Paper award</a>!
After two rejections from other conferences, and the fact that it only got
accepted to PETS thanks to a
<a href="https://petsymposium.org/experiment.php">consistency experiment</a> (one group of
reviewers accepted it, not the other), it was unexpected and nice. My talk was
recorded, you can watch it <a href="https://youtu.be/wjlAbh0qQQQ?t=54">here</a>.</em></p>
<p>Also, the "a-ha!" moment to get the generic result didn't happen when I
expected. It didn't click when I was spending hours working on it in the lab.
Instead, it struck in the shower, after spending a week of vacation without an
Internet connection. I strongly suspect this isn't a coincidence… Logging off
must have helped my brain be more relaxed and creative or something.</p>
<h1 id="what-comes-next">What comes next</h1>
<p>I'll try to write a similar blog post for each research paper I publish<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>. If
you're a researcher, I'd encourage you to do the same. The time it takes to
publish a write-up like this is negligible compared to doing the research and
writing the paper… And many more people will be able to read it!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>Emphasis on "try". When the paper is purely the result of research done at
Google, like <a href="https://ai.google/research/pubs/pub47664">this one</a>, there might
be complications. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Almost differential privacy2019-02-20T00:00:00+01:002020-03-05T00:00:00+01:00Damien Desfontainestag:desfontain.es,2019-02-20:/privacy/almost-differential-privacy.html<p>Publishing histograms without knowing the categories in advance: introducing (ε,δ)-differential privacy.</p><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p></small></p>
<hr>
<p><span class='lettrine'>L</span><strong>et's</strong> continue where we left off. In the <a href="differential-privacy-in-practice.html">last
article</a>, we saw how to publish histograms in a privacy-preserving way. Adding
noise to each count was enough to get ε-differential privacy. But we finished
with a puzzling statement: I mentioned that if you don't know the categories in
advance, the technique no longer works. In fact, the problem gets much trickier.
We'll even need to introduce a variant of the original definition! Let's dive
in.</p>
<h1 id="open-ended-survey-question">Open-ended survey question</h1>
<p>Let's say you're doing a survey where you asked people what's their favorite
color. Instead of giving them a list of fixed options, you let them write
whatever text they want. Lots of answers are going to be common colors: <em>blue</em>,
<em>green</em>, <em>pink</em>… But real-world data is noisy, and you're surely going to get
unpredictable answers. Some might be junk answers: people misunderstanding the
question, or trolling the survey. Other might simply be rare colors. You want to
publish a histogram of answers.</p>
<p>Let's use the same technique as before. What happens if we add <a href="https://en.wikipedia.org/wiki/Laplace_distribution">Laplace noise</a>
of scale <span class="math">\(1/\varepsilon\)</span> to each category? We need to compare the output of this
process for two databases that differ on a single element. There are two
possibilities.</p>
<h2 id="the-two-databases-have-the-same-categories">The two databases have the same categories</h2>
<p>If you're lucky, the two databases have the same categories. For example:</p>
<ul>
<li>In one, you got 10 <em>green</em> answers, 5 <em>red</em>, and 2 <em>yellow</em>.</li>
<li>In the other, you got 10 <em>green</em> answers, 5 <em>red</em>, and <strong>3</strong> <em>yellow</em>.</li>
</ul>
<p>Then, adding noise to each category works fine. The only difference is in the
<em>yellow</em> category. By adding noise, we hide the difference between the two
values, exactly like <a href="differential-privacy-in-practice.html">before</a>.</p>
<h2 id="the-two-databases-dont-have-the-same-categories">The two databases don't have the same categories</h2>
<p>This is where it gets trickier. For example:</p>
<ul>
<li>In one, you got 10 <em>green</em> answers, 5 <em>red</em>, 2 <em>yellow</em>.</li>
<li>In the other, you got 10 <em>green</em> answers, 5 <em>red</em>, 2 <em>yellow</em>, <strong>and one
<em>ultramarine</em>.</strong></li>
</ul>
<p>Let's see what happens if you add noise to both. Each column will end up with a
slightly different number than the real one. But there's something glaringly
obvious: <em>the categories are different</em>!</p>
<p><center>
<img alt="Histogram with three columns corresponding to colors, with noise" src="https://desfontain.es/privacy/images/3-color-noised-histogram.svg">
<img alt="Histogram with four columns corresponding to colors, with noise" src="https://desfontain.es/privacy/images/4-color-noised-histogram.svg">
</center></p>
<p>No need to squint at the numbers to notice the difference between these two
histograms! It's easy for an attacker to tell apart outputs with different
categories. We call this a <strong>distinguishing event</strong>: the attacker can learn with
100% certainty which database is the right one. Thus, the process is not
differentially private. How to fix this?</p>
<p>Maybe we could list all <em>possible</em> categories, and add noise to each of them,
including zero counts. Unfortunately, it's not as simple as that. In our
example, people can enter <em>anything</em>: there's an infinite number of
possibilities. The good news is that at the cost of a slight relaxation in our
privacy guarantee, we can overcome that problem.</p>
<h1 id="a-solution-thresholding">A solution: thresholding</h1>
<p>It's fairly difficult to make sure distinguishing events never happen<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>. Instead, we can settle for the next best thing: we prevent them from happening
<strong>most of the time</strong>. One way to do that is <strong>thresholding</strong>. Not only do we add
noise to each category, but we also remove all categories with low counts. Let's
say that our threshold is <strong>5</strong>. In the example above, we would end up releasing
only two categories:</p>
<p><center>
<img alt="Histogram with two columns corresponding to colors, with noise" src="https://desfontain.es/privacy/images/2-color-noised-histogram.svg">
</center></p>
<p>There's a price to that strategy: we're losing rare categories. In this example,
we didn't only drop the <em>ultramarine</em> category, but <em>yellow</em> as well. Any
category whose count is close to 5 (or less) has a significant chance of being
lost. Often, that's not a big problem: rare answers have a larger chance of
being meaningless.</p>
<p>That solution isn't perfect from a privacy perspective. For example, what if the
noise added to the ultramarine category is larger than 4? Then the total count
is 5 or more, we end up publishing this category, and it breaks differential
privacy. Fortunately, this doesn't happen too often: only 0.6% of the time with
Laplace noise of parameter <span class="math">\(1/\ln(3)\)</span>.</p>
<h1 id="tying-it-all-together-varepsilondelta-differential-privacy">Tying it all together: <span class="math">\((\varepsilon,\delta)\)</span>-differential privacy</h1>
<p>Our strategy is a little more complicated than before. We now have two
parameters.</p>
<ul>
<li>The <strong>amount of noise</strong> we're adding. Just like before, if we're aiming for
<span class="math">\(\varepsilon\)</span>-differential privacy most of the time, we need to add Laplace
noise of scale <span class="math">\(1/\varepsilon\)</span>.</li>
<li>The <strong>threshold</strong> we're using to drop rare categories, after adding noise. It
induces a natural trade-off. The bigger the threshold, you more data you lose…
But the bigger the threshold, the more you reduce the odds of having a
distinguishing event.</li>
</ul>
<p>Let's visualize this. For each threshold, what are the odds that by adding noise
to a category with count 1, you end up above the threshold? The following graph
assumes Laplace noise of parameter <span class="math">\(1/\ln(3)\)</span>. </p>
<p><center> <img alt="Graph showing the probability of a distinguishing event depending on
the threshold" src="https://desfontain.es/privacy/images/laplace-tail-depending-on-threshold.svg">
</center></p>
<p>Using a logarithmic scale, the graph is a straight line. That makes sense:
Laplace noise is a double exponential distribution.</p>
<p>Now, the choice of threshold is specific to the algorithm. For a different
algorithm, or a different noise function, the same threshold might have a
different effect. So it's not a good idea to use it directly to quantify
privacy. Instead, we use the <strong>odds of a distinguishing event</strong> as an additional
parameter to our modified definition.</p>
<h4 id="formal-definition">Formal definition</h4>
<p>From <span class="math">\(\varepsilon\)</span>-differential privacy, we get
<strong><span class="math">\((\varepsilon,\delta)\)</span>-differential privacy</strong>. This new definition is stricly
weaker than the original definition, and has a similar formulation. For all
databases <span class="math">\(D_1\)</span> and <span class="math">\(D_2\)</span> which differ in only one individual, and all sets <span class="math">\(S\)</span>
of outputs:</p>
<div class="math">$$
\mathbb{P}[A(D_1)\in S] \le e^\varepsilon\cdot\mathbb{P}[A(D_2)\in S]+\delta.
$$</div>
<p>The meaning of <span class="math">\(\varepsilon\)</span> is the same as <a href="differential-privacy-in-more-detail.html#quantifying">before</a>. The only new
element is the <span class="math">\(\delta\)</span>. It captures the odds that something goes wrong<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup>. In
our example above, <span class="math">\(\delta\approx0.006=0.6\%\)</span>. By using
<span class="math">\((\varepsilon,\delta)\)</span>-differential privacy, we're saying that the algorithm is
<em>almost</em> <span class="math">\(\varepsilon\)</span>-differentially private. And here, <em>almost</em> means <em>with
probability <span class="math">\(1-\delta\)</span></em>: the closer <span class="math">\(\delta\)</span> is to 0, the better.</p>
<h4 id="criticisms-of-the-definition">Criticisms of the definition</h4>
<p>As I said, you can see <span class="math">\(\delta\)</span> as the probability that something goes terribly
wrong. For a privacy definition, this seems like a bad thing to have. Consider
the following algorithm, which takes a database as input. With probability
<span class="math">\(1-\delta\)</span>, it returns 42. With probability <span class="math">\(\delta\)</span>, it returns the entire
database. Talk about a data leak! Still, this algorithm is
<span class="math">\((0,\delta)\)</span>-differentially private.</p>
<p>This example illustrates that this <span class="math">\(\delta\)</span> parameter allows for <em>catastrophic
failures</em><sup id="fnref:3"><a class="footnote-ref" href="#fn:3">3</a></sup>. Knowing this, you have two options.</p>
<ol>
<li>Either you work harder to predefine categories, or use more advanced
techniques, and only use "true" differential privacy.</li>
<li>Either you accept that bad things can happen, and try to limit the risk by
mandating a tiny <span class="math">\(\delta\)</span>.</li>
</ol>
<p>I'd argue that the second solution is not a bad choice. The probability of
getting hit by lightning in your lifetime is on the order of
<a href="https://www.weather.gov/safety/lightning-odds"><span class="math">\(10^{-4}\)</span></a>. The probability of a given bit in your RAM being
randomly flipped by a cosmic ray in one year is about <a href="https://stackoverflow.com/a/23587649"><span class="math">\(10^{-6}\)</span></a>. In
many situations, it's reasonable to consider these a negligible risk.</p>
<p>My perspective is that everything in data protection is about risk mitigation.
You'll never reduce the risk to 0. Even if you use "true" differential privacy,
your implementation might have critical bugs. Or you might get hacked, and your
entire anonymization strategy might become irrelevant. Or someone might drug you
and hit you with a <a href="https://xkcd.com/538">$5 wrench</a> until you give them your database.
What are the odds of this happening? If your <span class="math">\(\delta\)</span> is even smaller, it might
be an acceptable price to pay for more convenience.</p>
<h4 id="how-to-choose-delta">How to choose <span class="math">\(\delta\)</span>?</h4>
<p>Considering the catastrophic scenarios above, maybe our <span class="math">\(\delta\)</span> of 0.6% is a
bit too large to use everywhere. But what's a good number? A common option is to
pick a <span class="math">\(\delta\)</span> that is significantly smaller than <span class="math">\(1/n\)</span>, where <span class="math">\(n\)</span> is the total
number of people in the database. The reasoning goes as follows. Each person
has, in the worst case, a <span class="math">\(\delta\)</span> chance that their data leaks. So the total
odds that someone's data leaks is <span class="math">\(\approx n\delta\)</span>: we need to make sure that
this number is small enough<sup id="fnref:4"><a class="footnote-ref" href="#fn:4">4</a></sup>.</p>
<p>Luckily, in the problem above, you don't need huge thresholds to get tiny values
of <span class="math">\(\delta\)</span>. If you have a million users, and you want <span class="math">\(n\delta<0.1\)</span>, a
threshold of 15 is enough.</p>
<h4 id="cool-properties">Cool properties</h4>
<p><span class="math">\((\varepsilon,\delta)\)</span>-differential privacy has the same convenient properties
as differential privacy.</p>
<ul>
<li><a href="differential-privacy-awesomeness.html#composition">Composition</a>: suppose you have two <span class="math">\((\varepsilon,\delta)\)</span>-differentially
private mechanisms. Then, publishing the result of both satisfies
<span class="math">\((2\varepsilon,2\delta)\)</span>-differential privacy.</li>
<li><a href="differential-privacy-in-practice.html#post-processing">Post-processing</a>: suppose you have a <span class="math">\((\varepsilon,\delta)\)</span>-differentially
private mechanism. Then if you make its output go through a fixed
transformation, you still get <span class="math">\((\varepsilon,\delta)\)</span>-differential privacy.</li>
</ul>
<p>That means that most of what we learned in the <a href="differential-privacy-in-practice.html">simpler case</a> of predefined
categories still applies. You can round noisy values to integers without risk.
If the same person can be in multiple buckets, you can adapt the values of
<span class="math">\(\varepsilon\)</span> and <span class="math">\(\delta\)</span>. You can also compute sums, although you should be
careful in how you adapt the threshold when doing so.</p>
<h1 id="future-steps">Future steps</h1>
<p>With that, we covered the most frequent and easy use cases for differential
privacy. Next, we'll take a closer look at this new definition,
<span class="math">\((\varepsilon,\delta)\)</span>-differential privacy. We'll see that meaning of <span class="math">\(\delta\)</span>
is actually a little more subtle than explained in this post. And this is the
perfect excuse to introduce an important concept: the <a href="privacy-loss-random-variable.html">privacy loss random
variable</a>. Or you can also head over to the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> of
this blog post series to decide what to read next!</p>
<hr>
<p><small>Thanks to <a href="http://www.frankmcsherry.org/about/">Frank McSherry</a> and <a href="http://a3nm.net/">Antoine Amarilli</a> for their helpful
comments.</small></p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>But, as it turns out, not completely impossible. The "Improved Adaptative
Histogram" method described in
<a href="https://github.com/frankmcsherry/blog/blob/master/assets/Synth-SIGMOD.pdf">this paper</a>
does exactly that, even if the space of possible categories is infinite. I
don't know how it compares to the approach described in this post in terms of
data loss & truthfulness. It'd be interesting to figure out! <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>This intuition is technically incorrect, but it's a good first
approximation. Most people can understand the idea of "a small chance that
something goes wrong". The real interpretation is more complex, as explained
in the <a href="privacy-loss-random-variable.html">next article in the series</a>. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>For more fun examples, check out this
<a href="https://github.com/frankmcsherry/blog/blob/master/posts/2017-02-08.md">blog post</a>.
Its author is one of the original creators of differential privacy. I
recommend checking his other posts! <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:4">
<p>Note that this assumes independence between all the possible data leakage
events. This is wrong in general, but it's a good enough approximation in
practice. <a class="footnote-backref" href="#fnref:4" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Personal open access policy2018-12-28T00:00:00+01:002018-12-28T00:00:00+01:00Damien Desfontainestag:desfontain.es,2018-12-28:/privacy/personal-open-access-policy.html<p>How I contribute to a healthier model of scientific publishing.</p><p><strong>I</strong> do not provide free work for closed-access scientific venues.</p>
<p>By "free work", I mean peer review or organizational work, like participating in
program committees. By "closed-access", I mean "which retains exclusive rights
on papers, and in particular prevents authors from publishing them under a
license like <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/">CC BY-NC-SA</a>".</p>
<p>Now, a few answers to frequently asked questions. </p>
<h1 id="why">Why?</h1>
<p>My time, like yours, is precious. If I work on something without being paid for
it, it's because I believe that it is somehow beneficial to the world.
Scientific venues that do not allow authors to publish their work for free, and
forbid others to reuse and republish it, do not meet this requirement.
Scientific knowledge is meant to be universally accessible, well-organized and
useful. Barriers, especially financial ones, impede scholarly work and
discriminate against underfunded researchers and institutions.</p>
<p>Some closed-access venues definitely do some valuable things, e.g., bringing
scientific minds together and providing organizational work. But scientific
papers are the results of work done entirely by researchers. There is absolutely
no reason why they should be owned by a third party. If closed-access venues
shut down all at once tomorrow; the scientific community would quickly adapt,
and the change would overall be extremely positive.</p>
<p>Scientific venues are gradually moving towards open access, but this is taking
place very slowly. I wish to accelerate the process, and I think it is past the
point where it makes any sense to encourage the old model.</p>
<h1 id="what-about-submitting-papers">What about submitting papers?</h1>
<p>I try not to publish in closed-access venues. However, I do not make this a
matter of absolute personal policy, like for reviewing, for two reasons.</p>
<p>First, I cannot take that decision entirely on my own. Scientific papers are
typically collaborations. My coauthors should have a say in where we publish.
This is particularly important since the stakes can be much higher for them than
for me. I do not want to pursue a purely academic career, so publishing in a
less prestigious venue does not cost me much. My coauthors might feel very
differently, and I respect that.</p>
<p>Second, publishing a paper in a prestigious venue is not exactly free work. I'm
personally getting a lot out of the deal: recognition, personal gratification,
visibility of my work… This is very much in contrast to peer reviewing or
organizational work, which researchers are doing because they feel a moral
obligation to do so.</p>
<h1 id="what-about-the-reciprocity-rule">What about the reciprocity rule?</h1>
<p>Researchers do free work because they know how important it is for the
scientific community. The peer review process is crucial to publishing good
science, and the organizing work behind conferences and journals is extremely
valuable to researchers. So, if you publish scientific papers that are reviewed
by your peers, you should also review papers.</p>
<p>I understand the importance of this rule, but I choose to do my share of peer
reviewing in open access venues. This way, I not only contribute to my field of
research, but I also encourage a better, more equitable and accessible
publishing system.</p>
<p>Many people interpret the reciprocity rule <a href="https://twitter.com/TedOnPrivacy/status/1078071360256122880">differently</a>, and think that
researchers should review where they publish. I respectfully disagree, but I
acknowledge that my position might seem ethically problematic if you feel this
way. I think that the ethical problems of closed-access scientific publishing
far outweigh these concerns.</p>
<h1 id="what-about-gray-areas-like">What about gray areas, like…</h1>
<h4 id="postprints">… postprints?</h4>
<p>The only difference between the <a href="https://en.wikipedia.org/wiki/Postprint">postprint</a> and the final published version is
typically minor editing work, if any. I consider this distinction negligible.
Thus, it's fine if the postprint can be published in open access, even if the
publisher's version cannot.</p>
<h4 id="preprints">… preprints?</h4>
<p>The <a href="https://en.wikipedia.org/wiki/Preprint">preprint</a> version of a paper is essentially a draft. Once a paper has been
improved after incorporating the comments of peer reviewers, all preprints
published online should be replaced by the improved version.</p>
<p>Allowing preprints but not postprints to be published in open access encourages
people to read the preprint, when a better version exists. In the worst case,
the draft version contains factual errors that were fixed during peer review.
Not replacing the preprint by the postprint is counterproductive, and completely
nonsensical. The people providing reviewing work did so for free, and the result
of their work should not be owned by a third party. If only preprints can be
published in open access, then I do not consider the venue as open access.</p>
<h4 id="papers-accessible-online-but-not-under-a-permissive-license">… papers accessible online, but not under a permissive license?</h4>
<p>Some publishers will tell you that they will make your paper freely accessible
online through some portal, but will still forbid you to publish it elsewhere
under a permissive license. These promises are worthless — if publishers lie or
impose ridiculous conditions before giving people access to your papers, what
are you going to do? Sue them? Anyway, the fundamental problem is that
publishers should not own rights over scientific papers. They should not be able
to hinder the dissemination of knowledge they did not create. I do not consider
venues of these publishers to be open access.</p>
<h4 id="author-fees-for-open-access">… author fees for open access?</h4>
<p>Some conferences allow authors to pay an <a href="https://en.wikipedia.org/wiki/Article_processing_charge">additional fee</a> for their work to
be published in open access. Some fees are reasonable, on the order of 100€ or
less, but when big publishers use this model, these fees are generally much
higher: <a href="https://www.acm.org/publications/openaccess">≈$1000 for ACM</a>, <a href="https://open.ieee.org/index.php/for-authors/article-processing-charges/">≈$2000 for IEEE</a>, and up to
an eye-popping <a href="https://www.elsevier.com/about/policies/pricing">≈$6000 for Elsevier</a>. Needless to say, such
high fees are absolutely ridiculous compared to the actual costs to the
publisher.</p>
<p>Small fees might be acceptable, especially if they are mandatory and are only
there to recoup publisher costs. Larger fees, however, creates a system that is
unfair to researchers that cannot pay, and actively encourages them <em>not</em> to
make their research available in open access. It also takes financial resources
out of valuable research projects, and gives it to publishers instead, which are
providing a comparatively negligible value. If this is the only way that a venue
allows authors to publish their own papers, I do not consider it open access.</p>
<h4 id="embargo-periods">… embargo periods?</h4>
<p>This refers to a model where authors have to wait <a href="https://en.wikipedia.org/wiki/Embargo_(academic_publishing)">a certain time</a> before
publishing their own papers on open repositories. Before this delay, the paper
is only available behind paywalls or subscriptions. Again, it is unfair to
students and researchers that are not well-funded, and is an artificial and
arbitrary barrier to the dissemination of scientific knowledge. To me, venues
which operate under this model are not open access.</p>
<h4 id="nonprofit-organizations">… nonprofit organizations?</h4>
<p>Some organizations earn money via copyrighted academic papers, and are doing
good scientific things with this money. I understand how it came to be, but I
still think that blocking access to scientific knowledge is not an acceptable
way to fund such programs. We, as a society, should absolutely fund scholarly
institutions and education programs. But we should do so via other means. </p>
<p>I agree that there are significant ethical differences between for-profit
scientific publishers and nonprofit scholarly institutions. But a closed-access
venue will not get my support simply because it is organized by a nonprofit.</p>
<h4 id="books-or-other-works-that-need-heavy-editing">… books, or other works that need heavy editing?</h4>
<p>Paid editing work is either nonexistent or useless for scientific conferences
and journals. But the situation is very different for books or monographs, most
of which require professional editing before publication. This work should be
compensated, and copyright is one way of achieving this. I do not know this
space well. By default, my personal policy does not apply there.</p>
<h4 id="other-types-of-free-scientific-work">… other types of free scientific work?</h4>
<p>Some other types of free work are not explicitly covered by this policy.</p>
<ul>
<li>As I already mentioned, publishing my own papers is not free work. Similarly,
giving a talk at a conference is not necessarily free work either. It depends
on the message, the audience, whether expenses are reimbursed…</li>
<li>Course materials, and other physical artifacts from education and mentoring
efforts, should in principle also be freely accessible. But I could imagine
that in some cases, benefits that come from these efforts might outweigh the
ethical issues that I have with closed-access publishing.</li>
<li>There are certain types of work that I simply do not know enough to have an
opinion about: reviewing grant proposals, participating in <a href="https://en.wikipedia.org/wiki/Standards_organization">technical
standards organizations</a>, etc.</li>
</ul>
<p>For all those examples, I decide what I do on a case-by-case basis rather than
applying a blanket policy.</p>
<h1 id="why-are-you-publishing-this">Why are you publishing this?</h1>
<p>First, I want to draw a line in the sand for myself, and taking a public stance
is an excellent way to do so. Second, I hope to start a discussion, and maybe
get other people to consider adopting a similar policy.</p>
<p>If everyone stopped reviewing for closed-access venues, the entire scientific
publishing model would change for the better overnight. Refusing to do free work
will likely not hurt your career significantly, especially if you do not take a
public position like I do.</p>
<h1 id="why-arent-you-going-further">Why aren't you going further?</h1>
<p>My standards are relatively low: allowing authors to retain rights to their own
papers should really be the absolute minimum. I think that journals and
conferences should do more. For example, they should automatically publish all
papers under a permissive license like <a href="https://creativecommons.org/licenses/by/4.0/">CC BY</a>, rather than simply
allowing authors to do so.</p>
<p>I do not feel as strongly about this. For now, I'm okay with providing free work
to venues which simply say "sure, feel free to put your papers under a
permissive license yourself". It's good enough, and in some cases, it's the best
compromise that program committees for major conferences could negotiate with
publishers. I might reconsider this in the future.</p>
<h1 id="why-are-you-so-judgmental">Why are you so judgmental?</h1>
<p>I hope I did not come across this way, but in case I did, I want to make the
following explicit. I absolutely understand that you might not make the same
choices than I do. I do not think badly of anyone choosing to do otherwise. Part
of why I can afford to have this stance is because it is not costing me much. It
is not my place to judge what other people should be doing, especially people
who would be more personally impacted by such choices.</p>
<p>I do hope that more people try to accelerate the switch to open access. Tenured
professors have more freedom to refuse to encourage closed-access conferences,
even if these conferences are the most prestigious. People in positions of power
within closed-access venues might lobby to change publishing policies. I think
that they should, but I'm not in their shoes. To them, other things might seem
more important or urgent to work on. It is about values, and it is not my place
to tell you what your values should be. </p>
<h1 id="is-this-type-of-personal-stance-really-the-best-way-towards-open-access">Is this type of personal stance really the best way towards open access?</h1>
<p>No. The best way is through funding agencies. They should mandate that all
papers published using their grants must be published in open access. Laws
should automatically give researchers the right to do so without fear of
retaliation from publishers, no matter what the copyright agreement says. Some
countries are already doing that. It is great, and should be generalized
globally. These top-down mandates are the best path towards a world where 100%
of scientific papers are published under an open access license. They have much
more impact than personal policies like mine.</p>
<p>Also, if I already had some power within publishing institutions, I might try to
accomplish change from the inside instead. This might be the most efficient path
for someone already there. I am not in that position, though, and probably will
not be in the near future. So, I am doing what I can, at my level.</p>
<h1 id="can-i-try-to-change-your-mind">Can I try to change your mind?</h1>
<p>Sure. My contact info is at the bottom of the page. Note that I have already
heard the following points, and taken them under consideration.</p>
<ul>
<li>"You're going to make some people angry and possibly hurt your career."</li>
<li>"For-profit publishers are useful, here's a long list of valuable things they
do."</li>
<li>"It is unethical to publish in a venue but refuse to peer review for it."</li>
<li>"Publishers generally don't bother you when you upload your papers on your
personal website, so there's no pragmatic reason to care about these things."</li>
</ul>
<p>If you want to try and convince me that I'm doing something wrong, I encourage
you to bring new ideas to the discussion.</p>
<hr>
<p><small>Note: none of these ideas is originally from me. I wrote this after being
inspired by friends and colleagues who also chose not to support closed-access
venues with free work. I'm not reproducing names for privacy reasons (not
everyone is comfortable with taking a public stance), but most of the credit
goes to them.</small></p>Differential privacy in practice (easy version)2018-11-22T00:00:00+01:002019-02-20T00:00:00+01:00Damien Desfontainestag:desfontain.es,2018-11-22:/privacy/differential-privacy-in-practice.html<p>How to add differentially private magic to your statistics, in the easy cases: counts, sums, averages, histograms…</p><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p></small></p>
<hr>
<p><span class='lettrine'>P</span><strong>reviously</strong>, we saw that differential privacy
was <a href="differential-privacy-awesomeness.html">pretty awesome</a>, and we looked at the <a href="differential-privacy-in-more-detail.html">formal definition</a>. Now, how do we
obtain it in practice? Let's start with the basics.</p>
<h1 id="counting-unique-users">Counting unique users <a name="counting-unique-users"></a></h1>
<p>Suppose you have a database, and you want to publish how many people in there
satisfy a given condition. Say, how many have green eyes? Even if you have many
people in your database, you can't just publish the true answer. Let's take a
moment to understand why. </p>
<p>With differential privacy, we assume that the attacker knows <em>almost all
elements</em>. They only have uncertainty about their target. Say they want to know
whether their target has green eyes. If you output the real number <span class="math">\(k\)</span>, they can
compare it with the number of people with green eyes among the people they know.
If it's <span class="math">\(k-1\)</span>, then the target has green eyes. If it's <span class="math">\(k\)</span>, then the target does
not.</p>
<p>So, what do we do? We compute the exact answer, and we add <em>noise</em>. This noise
will come from a probability distribution called the <a href="https://en.wikipedia.org/wiki/Laplace_distribution"><em>Laplace
distribution</em></a>. This distribution has a parameter, its <em>scale</em>, which
determines how "flat" it is. It looks like this:</p>
<p><center>
<img alt="Graph showing a Laplace distribution with scale 1/ln(3), centered on 0" src="https://desfontain.es/privacy/images/laplace-ln-3.svg">
</center></p>
<p>So, to get <span class="math">\(\varepsilon\)</span>-differential privacy, we pick a random value according
to <span class="math">\(\text{Laplace}(1/\varepsilon)\)</span>, and we add this noise to the real value. Why
does it work? Let's look at the distribution of the number we return, depending
on whether the true count is <span class="math">\(k=1000\)</span> (blue line, the target doesn't have green
eyes) or <span class="math">\(k=1001\)</span> (yellow line, the target has green eyes).</p>
<p><center>
<img alt="Graph showing two Laplace distributions with scale 1/ln(3), centered on 1000 and 1001" src="https://desfontain.es/privacy/images/two-laplace-ln-3.svg">
</center> </p>
<p>Let's say the real number is <span class="math">\(k=1001\)</span>, and after adding noise, we published
<span class="math">\(1003\)</span>. Let's put ourselves in the attacker's shoes. What's the likelihood that
the original number was <span class="math">\(1001\)</span> vs. <span class="math">\(1000\)</span>? The hypothesis "<span class="math">\(k=1001\)</span>" is a bit
more likely: generating a noise of <span class="math">\(2\)</span> is more likely than a noise of <span class="math">\(3\)</span>. How
much more likely? It turns out that the <em>ratio</em> between these likelihoods is…
<span class="math">\(e^\varepsilon\)</span>! So the <a href="differential-privacy-in-more-detail.html#definition">ratio of probabilities</a> of differential privacy
is satisfied.</p>
<p>This works no matter what the output is: the ratio will always be between
<span class="math">\(e^\varepsilon\)</span> and <span class="math">\(e^{-\varepsilon}\)</span>. If you want to double-check, you can
either verify it on the graph, or do the <a href="https://en.wikipedia.org/wiki/Laplace_distribution">math</a>.</p>
<h1 id="counting-things">Counting things <a name="counting-things"></a></h1>
<p>OK, so counting unique users was pretty easy. Counting things must also be
straightforward, right? Let's say you have a database of suggestions that people
sent to your company using a feedback form. You want to publish the number of
suggestions you received on a given day. Meanwhile, the attacker wants to get an
idea of how many complaints their target published.</p>
<p>What's different about the previous scenario? Can't we just add noise picked
from <span class="math">\(\text{Laplace}(1/\varepsilon)\)</span> and get <span class="math">\(\varepsilon\)</span>-differential privacy?
There's a catch: what if someone sent more than one complaint during one day?
Let's say someone was super unhappy and sent <em>five</em> complaints. The other 1000
customers sent one complaint each. The influence of this one disgruntled
customer will be larger than before. The two distributions now look like this:</p>
<p><center>
<img alt="Graph showing two Laplace distribution with scale 1/ln(3), centered on 1000 and 1005" src="https://desfontain.es/privacy/images/two-laplace-ln-3-interval-5.svg">
</center></p>
<p>The difference between the curves is much larger than before. Their ratio is at
most <span class="math">\(e^{5\varepsilon}\)</span>, so using a parameter of <span class="math">\(1/\varepsilon\)</span> only gives
<span class="math">\(5\varepsilon\)</span>-differential privacy. To fix this, we need to add more noise. How
much more? It depends on the <em>maximum contribution</em> of one individual user. If
the maximum amount of complaints in one day is 5, you must add 5 times the
amount of noise. In this example, using <span class="math">\(\text{Laplace}(5/\varepsilon)\)</span> would
give you <span class="math">\(\varepsilon\)</span>-differential privacy.</p>
<p><a name="laplace"></a>
<center>
<img alt="Graph showing two Laplace distribution with scale 5/ln(3), centered on 1000 and 1005" src="https://desfontain.es/privacy/images/two-laplace-5-ln-3.svg">
</center></p>
<p>Note that you can't fully automate this process: you need to know what the
largest contribution can be. A human, with some knowledge over the process, must
make a judgment call. In our case, this could be "users won't post more than
five complaints per day".</p>
<p>What happens if that judgment call is wrong, and a user later decides to post 10
complaints in one day? To preserve the desired level of privacy, you need to
<em>clamp</em> all values to the estimated maximum. In other words, for this outlier
user, you would only count 5 complaints in the non-noisy sum. </p>
<p>This process can introduce unexpected bias in the data. So, be careful when
estimating the largest contribution! If clamping only happens very rarely, you
should be fine.</p>
<h1 id="summing-or-averaging-numbers">Summing or averaging numbers</h1>
<p>Let's say each of your users gives your customer service a rating, between -10
and 10. You want to release the average rating. Computing an average is pretty
much the same as computing a sum — add all ratings, then divide by the number of
users<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>. So, what do we do to the sum to achieve differential privacy?</p>
<p>Among all rating options, we only have to consider the worst possible case. How
far can the two noise curves be from each other? If the values are all between
-10 and 10, the greatest possible difference is <span class="math">\(10-(-10)=20\)</span>. It happens when
the attacker tries to determine whether a user voted -10 or 10.</p>
<p>Like in the previous example, you have to add noise of <span class="math">\(Laplace(20/\varepsilon)\)</span>
to get <span class="math">\(\varepsilon\)</span>-differential privacy. And just as before, you need to check
that each value is between your theoretical minimum and maximum. If you find an
anomalous value, e.g. lower than the minimum, you need to <em>clamp</em> it to the
minimum before adding it to the sum.</p>
<p>In some cases, estimating these minimum or maximum values is difficult. For
example, if you're computing the average salary in a large group of people, how
should you estimate the upper salary limit? I don't see that problem as a
usability flaw of differential privacy. Rather, it suggests that averages are
not a meaningful metric in the presence of outliers. Removing these outliers is
a good idea for both accuracy and privacy :-)</p>
<h1 id="releasing-many-things-at-once">Releasing many things at once <a name="many-things"></a></h1>
<p>OK. What if you don't want to release only one statistic, but many of them? Can
you add noise to each of them and be fine? Well… it depends. The main question
you have to ask yourself is: <em>what is the maximum influence of one individual</em>?
There are two distinct possibilities.</p>
<h2 id="statistics-are-about-different-people">Statistics are about different people</h2>
<p>Suppose you want to release the number of users you have depending on their age
ranges: 20-29, 30-39, 40-49, etc. <a name="histogram"></a></p>
<p>Each user will have an influence in at most <em>one</em> of these categories. Either
someone is in a given age range, either they're in another one. This situation
often appears when you're trying to release a <em>histogram</em>:</p>
<p><center>
<img alt="Histogram of fake data showing a number of people by age range" src="https://desfontain.es/privacy/images/histogram.svg">
</center></p>
<p>When you're in this case, you can safely add Laplace noise of scale
<span class="math">\(1/\varepsilon\)</span> to each bucket count. There is no problematic interaction
between buckets. Releasing the entire histogram is still
<span class="math">\(\varepsilon\)</span>-differentially private.</p>
<p><center>
<img alt="Noised version of the previous histogram, some bars have changed slightly" src="https://desfontain.es/privacy/images/noised-histogram.svg">
</center></p>
<p>Pretty easy, right? Note that this histogram looks a bit weird: the counts are
not integers, and one count ended up being negative! We can make it a bit less
suspicious, by rounding all counts and replacing all negative numbers by 0.</p>
<p><center>
<img alt=""Cleaned" version of the previous histogram, all values are now positive integers" src="https://desfontain.es/privacy/images/cleaned-noised-histogram.svg">
</center></p>
<p><a name="post-processing"></a>This type of <em>post-processing</em> is allowed, thanks
to a nifty property of differential property. If you take differentially private
data, and make it go through a fixed transformation, you still get differential
privacy<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup>. Convenient!</p>
<h2 id="one-person-is-in-multiple-statistics">One person is in multiple statistics <a name="multiple-statistics"></a></h2>
<p>OK, what if you're releasing multiple statistics, but this time, they might all
be about the same user? Let's say that you want to publish how many of your
users…</p>
<ul>
<li>are younger than 35;</li>
<li>are using an iOS device;</li>
<li>are colorblind;</li>
<li>have started using your app less than a month ago.</li>
</ul>
<p>The same user could be in all those categories! In this scenario, you can't add
Laplace noise of scale <span class="math">\(1/\varepsilon\)</span> to each count and get
<span class="math">\(\varepsilon\)</span>-differential privacy. Instead, you have to consider each count as
a separate data release. Thus, if you have <span class="math">\(C\)</span> different counts, you have to add
Laplace noise of scale <span class="math">\(C/\varepsilon\)</span> to each of them. Each independent release
will be <span class="math">\(\varepsilon/C\)</span>-differentially private. And we can now use the
<a href="differential-privacy-awesomeness.html#composition">composition</a> property of differential privacy! This allows us to conclude that
the entire release is <span class="math">\(\varepsilon\)</span>-differentially private.</p>
<p>This works for any kind of statistics, not just unique counts. Want to release
several pieces of information? Count the maximum influence of one single person,
and "split" your <span class="math">\(\varepsilon\)</span> between each data release. This <span class="math">\(\varepsilon\)</span> is
called your <em>privacy budget</em>: you choose <span class="math">\(\varepsilon_1\)</span>, …, <span class="math">\(\varepsilon_C\)</span>
whose sum is <span class="math">\(\varepsilon\)</span>, and you release the <span class="math">\(i\)</span>-th statistic with
<span class="math">\(\varepsilon_i\)</span>-differential privacy. This solution is more flexible than simply
using <span class="math">\(\varepsilon/C\)</span>-differential privacy on each statistic. If one statistic
is more important, or more sensitive to noise, you can attribute more budget for
it. The larger the budget portion, the lower the noise. Of course, you will have
to add more noise to the other values.</p>
<h1 id="traps-to-avoid">Traps to avoid</h1>
<p>The mechanisms we saw today are pretty straightforward. But anonymization is
full of traps: there are still a number of things that can go wrong.</p>
<ul>
<li>When summing or averaging numbers, <em>clamping</em> them to the minimum and maximum
is essential. Otherwise, all guarantees fly out the window.</li>
<li>Pay attention how you implement this clamping in practice. Special values like
<code>NaN</code> can lead to surprising behavior.</li>
<li>Of course, you're not allowed to cheat. You have to choose your privacy
strategy in advance, then apply it, and release the noisy data. You can't
double-check that it's accurate enough. Otherwise, you skew the randomness,
and you lose the privacy guarantees.</li>
<li>When releasing histograms, it's important to choose each category <em>in
advance</em>. If you have to look at your data to know what your categories are,
you have to use more subtle methods. </li>
</ul>
<p>This last point is often a problem in practice. For example, say that you want
to count how many times each word was used in customer complaints. You can't
make a definite word list: people could use words that you didn't predict, or
make typos. In that case, the technique I described for histograms doesn't work. </p>
<p>Why doesn't it work? How to fix this? All this, and more, in the <a href="almost-differential-privacy.html">next article</a>!
Or you can also head over to the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> of this blog post
series to pick what to read next.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>If you want to be extra pedantic, you might also want to add noise to your
total number of users. That depends on the flavor of definition that you
choose. I'm not going to that level of detail here, and you probably shouldn't
either. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>If you're wondering why this is true, here's a super short proof. If
there was a post-processing function that would break the differential privacy
property… The attacker could run it too, and distinguish between two outcomes.
But it's impossible, because differential privacy forbids it :-) <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>« So, how does your part-time PhD arrangement actually work? »2018-09-01T00:00:00+02:002018-09-05T00:00:00+02:00Damien Desfontainestag:desfontain.es,2018-09-01:/privacy/part-time-phd.html<p>I got this question many times. So I thought I'd answer it, along with other frequent questions about this arrangement.</p><p><strong>Since</strong> December 2016, I work as an engineer for Google 50% of my time, and I
do a PhD at the ETH in Zürich on the remaining 50%. Recently, some people have
asked me a bunch of questions about this arrangement. So here's a blog post
answering them, in no particular order. </p>
<p>Note: everything in this post is only from my own experience, and represents in
no way the opinion of my employers.</p>
<h1 id="so-google-does-part-time-phds-hm">So Google does part-time PhDs, hm?</h1>
<p>Not really.</p>
<p>When I first thought about this project, I tried to look for Googlers who did
this before me. I couldn't find anyone. Some folks studied (for Bachelor's,
Master's degrees, etc.) while working at Google. Some got hired before
completing their PhD, and finished it while working full-time or 80%. But I
couldn't find anybody who did a PhD from start to finish while working there.</p>
<p>So I sort of made up my own project. After a lot of work, I got it approved, and
now I'm in the middle of it. I can't detail the approval process publicly, but
this was definitely neither standard nor easy. One important takeaway: I
couldn't have done this as a new hire. Being a productive, full-time engineer
for a couple of years gave me the credibility I needed.</p>
<p>In short: it is possible to negotiate a part-time PhD arrangement when you work
at Google. But it depends on a lot of factors, many of which you have little to
no influence over. You should not base your career plans on the assumption that
you'll be able to do this at Google.</p>
<h1 id="why-would-you-do-this">Why would you do this?</h1>
<p>Because I was grumpy and unhappy with the other options :D</p>
<p>Many academics are doing research that has no real-world impact. Some of them
don't care, and that's fine. But even for the others, it's difficult to figure
out what problems are worth looking at. When you start doing research, you don't
know what's impactful and what's useless. So, you pick a novel and interesting
topic, and you hope it'll end up being useful. Then, before your realize it, you
get corrupted by the many things that are <a href="https://a3nm.net/work/research/wrong/">wrong with
academia</a>. A few more years, and you
completely forget that usefulness was ever a goal<sup id="fnref:0"><a class="footnote-ref" href="#fn:0">1</a></sup>.</p>
<p>At the same time, I thought that the tech industry is in general not a great
place to do research. Your job is to fix problems that your company has.
Figuring out whether you're the first to solve them, generalizing the solution,
publishing it for more people to use, educating yourself on vaguely related
work… That's not really what they pay you for. If you're an experienced
researcher working for a tech company, you might get this stuff done. You know
the value of such work, and you're able to do it and sell it to higher-ups. When
you don't know how to do research yet, it's not the same story.</p>
<p>My solution was to do both things at once. First, use my industry job to figure
out which problems are worth solving. Then, solve them thoroughly, not only in
the context of one team inside one company. Also, share the results with the
world. And let's be honest: in the process, I also hoped to become an expert in
the field =) </p>
<p>In short, I wanted to get all the good parts of a PhD, while avoiding the
standard traps of academic research. I didn't want to stop having a constant
stream of real-world problems to solve<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">2</a></sup>.</p>
<p><small>plus, what could possibly go wrong, right</small></p>
<h1 id="how-do-you-organize-your-time">How do you organize your time?</h1>
<p>I usually work one week at Google, and the next week at the university.</p>
<p>During the first few months, I was pretty strict with my schedule. It worked
okay. Nowadays, I often have research meetings during my Google time, or
vice-versa. It makes things easier for everyone, and doesn't hurt my
productivity too much, especially for early or late meetings. In addition, I
sometimes switch entire weeks, for example when I travel abroad to do Google
work for several weeks in a row.</p>
<h1 id="what-about-context-switching-costs">What about context switching costs?</h1>
<p>They're not a huge problem for me. Context switches affect different people
differently. Like most people, I can't get any serious work done in a half-hour
break between two meetings. But it's not that painful for me to work on a
different thing than the day/week before.</p>
<p>In my humble opinion, the ability to work on several things in parallel is a
core engineering skill. I suspect it's a crucial skill for many other jobs. You
should aim to be good at it no matter what you do. Having two jobs makes it
more obvious, but not fundamentally different.</p>
<p>[To be clear: I'm encouraging you to try and get better at this if you can. But
like any other skill, this is workable for some people and not so much for
others. Neurodiverse folks might have a harder time. If you're in a position of
leadership, build teams that accommodate specific needs.]</p>
<h1 id="what-are-the-challenges">What are the challenges?</h1>
<p>Some things don't get divided by two when you start working part-time.</p>
<p>The obvious one is overhead. The number of Google emails didn't get halved when
I started working half-time. Expense reports, performance reviews,
administrative nonsense… Mandatory and pointless tasks cost you twice as much
productivity.</p>
<p>Second, anxiety and impostor syndrome. I was prone to these when I started
working at Google, but I sort of learned to live with them over time. They came
back full-force when I started my PhD, and not only because of the unique
challenges of research work<sup id="fnref:3"><a class="footnote-ref" href="#fn:3">3</a></sup>. I instinctively compared myself to peers a
lot — I always did. But of course, working half-time, I couldn't get the same
amount of work done, so I felt shitty about it.</p>
<p>Finally, I got (at least) twice as many interesting work-related opportunities.
And since I had a tendency to work too much and accept too many responsibilities
and tasks… This definitely got worse.</p>
<h1 id="how-do-you-deal-with-those">How do you deal with those?</h1>
<p>For overhead, aggressive email filtering and management is part of the solution.
I also learned to notice when my brain can't handle intellectual, creative, or
stressful work anymore. When I catch myself procrastinating, I don't force
myself to go back to the thing I was supposed to do. Instead, I do something
that doesn't require as much brain, but is still productive. Answering emails,
solving administrative stuff, writing blog posts… This way, I don't waste too
much time, and the overhead doesn't cannibalize my real work too much.</p>
<p>To avoid overworking, I have some practical rules. I don't work on evenings,
weekends or vacation days. I use a different laptop for my Google work (where I
get most interruptions). I lock myself out of my work email outside office
hours. I force myself to do sports. Those "practical" tricks help. But of
course, the psychological aspect is the difficult part…</p>
<p>I try to remind myself that I'm not responsible for the stuff that I'm not in
charge of. Most importantly, I started saying no to things. This is <strong>so
difficult</strong> for me. It's not that my colleagues (or manager, advisor…) demand
too much of me — they're awesome and respect boundaries. But there are always
things to be done, ideas to research, stuff to try out, people to help and
collaborate with. Be smarter than me: learn how to set limits for yourself, and
say no to people, before starting a project like this.</p>
<p>On anxiety & impostor syndrome, here I some things I found helpful. </p>
<ul>
<li>Getting feedback from peers as often as possible. That helped me a lot. I tend
to have unrealistic expectations for myself. Having an external perspective
prevents me from getting stuck into weird self-critical loops.</li>
<li>When considering my own recent work, I try to look back a couple of months
rather than a couple of days/weeks. Being unproductive for a few days feels
much worse when you're only doing this 50% of your time. Even though you know
it's normal and happens to everyone, <em>especially</em> in research! Averaging over
larger time periods always feels better.</li>
<li>I started to talk to a mental health professional, and it was one of the best
decisions I took recently. I very much recommend doing so, if that's an option
for you. Even if you feel okay, it's never a bad idea to monitor how well your
brain is doing (and it's difficult to do on your own).</li>
</ul>
<p></p>
<h1 id="that-sounds-bad-is-it-that-bad">That sounds bad. Is it that bad?</h1>
<p>It's challenging, so I would not go around recommending random people do the
same. But no, it's not that bad =) I definitely don't regret starting this. Here
are some of my favorite things about this arrangement.</p>
<ul>
<li>I'm doing cool research that I know is impactful. When you're in academia, the
main indicator you have for that are publications and grants. My research ends
up changing policies and practices of a massive company. I'm improving the
level of data protection for billions of users! It definitely makes it easier
not to care too much about paper rejections ^^</li>
<li>I'm working my way towards becoming an expert in my field. Don't get me wrong:
I'm definitely not there yet. But I'm learning every day. I'm getting a deep
knowledge of some things, and a reasonably solid overview of many other
things.</li>
<li>I became the go-to person people in my Google team to answer research-related
questions. What's the state of the art for a given problem in my field, what
are known impossibility results, how to publish papers and engage with the
community… I can provide a unique perspective, which is rewarding and useful.</li>
<li>In my research lab, I also sort of became the go-to person to answer
industry-related questions. I help colleagues understand the applicability of
their work, and I can suggest potentially impactful research directions.</li>
<li>Talking about the same problems in both places leads to very different
discussions. I get ideas from Google, bounce them off university colleagues,
or vice-versa. The diversity of viewpoints I get this way is always fruitful
and fascinating. </li>
</ul>
<p>On a day-to-day basis, the main benefit is getting progress from both sides at
once. When I use my research at Google, my productivity basically doubles. It's
the same way when I can use my Google work to write papers or advance my
research. And it happens quite often! More than often enough to compensate for
the overhead problem I mentioned earlier.</p>
<h1 id="whats-the-best-thing-about-the-arrangement">What's the best thing about the arrangement?</h1>
<p>I was anticipating all the good bits described above. It was very nice to see
that I wasn't too optimistic, but this wasn't a big surprise. Those were why I
wanted to do a PhD in the first place =)</p>
<p>What I didn't expect, though, was the <em>perspective</em> it gave me. Let me explain.
The mindset in the tech industry is <strong>so</strong> different from academia. I know, it's
obvious, but… this rift was much bigger than what I imagined. The two worlds
have completely different values, social structures, incentives… And that
translates to incredibly distinct cultures and practices.</p>
<p>When you get into one of these worlds, you tend to adopt its implicit culture
and assumptions. The assimilation is real, even if your team is diverse and
inclusive. Want to get an intuitive idea of how strong this phenomenon is? Go
talk to a people who worked at the same big tech company for a while, or to
tenured professors.</p>
<p>Being in two places at once forces you to question the assumptions and cultural
baggage. I think it's pretty unique. I don't approach problems the same way as
when I was working full-time (at Google, or in research internships). The way I
judge my own work also changed a lot. I don't know how to describe it well, but
I feel much more "free" as a result.</p>
<h1 id="so-i-should-do-the-same-thing-you-did-right">So I should do the same thing you did, right?</h1>
<p>Before I answer that question with an enthusiastic "Yes! :D", I want to mention
a couple of more things. You should definitely be aware of them before making
this kind of decision.</p>
<h4 id="alternative-options">Alternative options</h4>
<p>Why are you interested in doing a part-time PhD? Think long and hard about this
question. In particular, consider all <em>other</em> options that achieve the same
goal.</p>
<p>Want to become an expert in a given field? There are plenty of ways of doing
that. Read books or scientific papers<sup id="fnref:4"><a class="footnote-ref" href="#fn:4">4</a></sup>. Attend courses, in-person or online.
Get a personal project to practice what you're learning. This will likely get
you to the level you want to be, faster and with less headaches.</p>
<p>Want the qualification because it's a shiny thing to put on your resume? In
computer science? Honest career advice: don't bother. Three years working in the
tech industry will look better on your resume. (And you'll also make more
money.) Relevant experience with personal or open-source projects can also have
a big impact.</p>
<p>Want a career in academia? Or maybe you're not sure, and you want to it figure
out? Doing a PhD full-time is the obvious option. You should also do that if you
need a PhD to get to your career goals in your field. It's less risky than my
part-time arrangement, and easier to organize<sup id="fnref:5"><a class="footnote-ref" href="#fn:5">5</a></sup>.</p>
<p>Want to learn how to do research? If you're working in a big tech company, there
might be more research-y teams to switch to. Maybe you can get mentorship from a
colleague who used to work in academia? Or collaborate with an external
professor on specific projects? If you're not sure that you want to spend a
large part of your time doing research, but want to try it out, this can be a
solid option.</p>
<h4 id="public-awareness-message">Public awareness message</h4>
<p>Finally, I want to insist on one last thing: my experience was very much
influenced by a lot of luck, privilege, and other external factors. If you
attempt the same thing, you might run into hurdles that I didn't have. To
illustrate that point, I made a list of things that I feel lucky about.</p>
<ul>
<li>I'm a cis white dude, so when I was advocating for my weird new kind of PhD
arrangement, people took me seriously by default. Ha! Grave mistake.
Nonetheless and in all seriousness, I'm sure it helped me get what I wanted.</li>
<li>I had a previous experience of research (via internships during my Master's
degree). Because of that, I already knew that I liked this type of work, which
made the project less risky. It also gave me credibility when I was trying to
find an advisor.</li>
<li>I don't have kids, strong family obligations, or serious health issues. The
time management aspect of things would surely be much harder otherwise.</li>
<li>At Google, my manager was awesome and supportive of my project from day one. I
couldn't have gone through the approval process without her. Many other
colleagues were also kind enough to spend time and resources to help me at the
time.</li>
<li>My current Google manager and my PhD advisor both give me a ton of freedom. To
a large degree, I can choose what to work on, which allows me to make sure
both my two jobs stay aligned. I've met people who weren't so lucky, and who
had awful part-time PhD experiences as a result.</li>
<li>Both of my jobs are a 15-minute bike ride away from where I live. Things would
be more difficult if I had a terrible commute.</li>
</ul>
<p>This list is most likely incomplete.</p>
<h1 id="i-have-other-questions">I have other questions!</h1>
<p>Fire away! Disclaimers: I can't answer much about Google processes if you're not
working there. And I don't guarantee a quick response time. But don't hesitate
to ask! My contact info is below =)</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:0">
<p>This does not accurately represent my current opinion of academia. At the
time though, it was a real fear of mine. Nowadays, I have a more moderate view
on the matter. Some academics manage to produce consistently impactful work. <a class="footnote-backref" href="#fnref:0" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:1">
<p>We have a <em>lot</em> of those at Google, and in particular in the privacy team. We're recruiting, by the way. Drop me a line if you're interested or would like more information =) <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>I might write about these later, when I have a bit more experience, and a
clearer picture of what they are exactly. <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:4">
<p>Scientific papers are awfully opaque when you start reading them, but it
quickly gets much easier. You'd be surprised! Also, if you don't have access
to scientific papers because of paywalls, you should check out
<a href="https://en.wikipedia.org/wiki/Sci-Hub">Sci-Hub</a>. <a class="footnote-backref" href="#fnref:4" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:5">
<p>Also, don't fall for the sunk cost fallacy if it turns out not to be your
thing. Quitting a PhD is okay. If you don't like it, go do things that make
you happy instead. You won't be the first nor the last, and the experience
will still be valuable. <a class="footnote-backref" href="#fnref:5" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
</ol>
</div>Differential privacy in (a bit) more detail2018-08-16T00:00:00+02:002019-02-20T00:00:00+01:00Damien Desfontainestag:desfontain.es,2018-08-16:/privacy/differential-privacy-in-more-detail.html<p>Why does differential privacy work so well? Let's look at it more closely.</p><p><small>
<span class='notlettrine'>T</span>his post is part of a <a href="friendly-intro-to-differential-privacy.html">series on differential
privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see the other
articles!</p>
<p></small></p>
<hr>
<p><span class='lettrine'>A</span><strong>s</strong> I mentioned in the <a href="differential-privacy-awesomeness.html">previous article</a>,
differential privacy is pretty awesome. If I did a good job, you're now
wondering what the <em>real</em> definition looks like. So in this post, I will go into
a bit more detail into what differential privacy actually means, and why it
works so well. There will be some math! But I promise I will explain all the
concepts I use, and give lots of intuition.</p>
<h1 id="the-definition">The definition</h1>
<p>We saw that a process satisfies differential privacy if its output is basically
the same if you change the data of one individual. And by "basically the same",
we meant "the probabilities are close".</p>
<p><center>
<svg
role="img"
aria-label="Diagram showing two fake "Guess Who" boards, the second one having one of the people missing. Each board, representing a database, has an arrow going to a silly "magic" gif, and this has an arrow going to a cloud labeled "output". A double arrow labeled "basically the same" points to the two outputs."
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
version="1.1"
viewBox="0 0 734.41882 358.09845"
stroke-miterlimit="10"
id="svg4123"
sodipodi:docname="full-diagram.svg"
inkscape:version="0.92.2pre0 (973e216, 2017-07-25)"
width="734.41882"
height="358.09845"
style="fill:none;stroke:none;stroke-linecap:square;stroke-miterlimit:10">
<title>Same diagram as before, duplicated, with the bottom line missing one person in the database. A double arrow labeled "basically the same" points to the two outputs.</title>
<desc>Same diagram as before, duplicated, with the bottom line missing one person in the database. A double arrow labeled "basically the same" points to the two outputs.</desc>
<metadata
id="metadata4129">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
<dc:title></dc:title>
</cc:Work>
</rdf:RDF>
</metadata>
<defs
id="defs4127">
<clipPath
id="p.3-4">
<path
d="M 0,0 H 275 V 252 H 0 Z"
id="path4077-0"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
</defs>
<sodipodi:namedview
pagecolor="#ffffff"
bordercolor="#666666"
borderopacity="1"
objecttolerance="10"
gridtolerance="10"
guidetolerance="10"
inkscape:pageopacity="0"
inkscape:pageshadow="2"
inkscape:window-width="1918"
inkscape:window-height="1078"
id="namedview4125"
showgrid="false"
inkscape:zoom="1.3111111"
inkscape:cx="532.33439"
inkscape:cy="127.30279"
inkscape:window-x="0"
inkscape:window-y="0"
inkscape:window-maximized="0"
inkscape:current-layer="g4121"
fit-margin-top="0"
fit-margin-left="0"
fit-margin-right="0"
fit-margin-bottom="0"
viewbox-x="82"
viewbox-width="650" />
<clipPath
id="p.0">
<path
d="M 0,0 H 960 V 720 H 0 Z"
id="path4036"
inkscape:connector-curvature="0"
style="clip-rule:nonzero" />
</clipPath>
<g
clip-path="url(#p.0)"
id="g4121"
transform="translate(-76.086586,-124.34907)">
<path
d="M 0,0 H 960 V 720 H 0 Z"
id="path4039"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 76.08921,124.34908 h 223.2992 V 267.11813 H 76.08921 Z"
id="path4041"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<g
transform="matrix(0.22307612,0,0,0.22307664,76.089214,124.34908)"
id="g4048">
<clipPath
id="p.1">
<path
d="M 0,0 H 1001 V 640 H 0 Z"
id="path4043"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
<image
clip-path="url(#p.1)"
width="1001"
height="640"
x="0"
y="0"
preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/quiestce.jpeg"
id="image4046"
style="fill:#000000" />
</g>
<path
d="M 76.086586,339.67847 H 299.38579 V 482.44751 H 76.086586 Z"
id="path4050"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<g
transform="matrix(0.22307612,0,0,0.22307664,76.086589,339.67847)"
id="g4057">
<clipPath
id="p.2">
<path
d="M 0,0 H 1001 V 640 H 0 Z"
id="path4052"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
<image
clip-path="url(#p.2)"
width="1001"
height="640"
x="0"
y="0"
preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/quiestce.jpeg"
id="image4055"
style="fill:#000000" />
</g>
<path
d="m 604.81974,178.70273 v 0 c -1.51233,-10.41394 3.45307,-20.72311 12.78925,-26.55296 9.33618,-5.82984 21.40576,-6.15791 31.08722,-0.84502 v 0 c 3.42944,-6.0551 9.70636,-10.23573 16.93207,-11.27733 7.2257,-1.04159 14.55151,1.17818 19.76159,5.98789 v 0 c 2.92145,-5.48995 8.65778,-9.17848 15.1734,-9.75675 6.51569,-0.57824 12.88855,2.03563 16.85718,6.9141 v 0 c 5.27802,-5.81934 13.67554,-8.2695 21.55884,-6.29033 7.8833,1.97917 13.83649,8.03222 15.28363,15.53996 v 0 c 6.46649,1.65271 11.8529,5.85413 14.76764,11.51872 2.91473,5.66462 3.07178,12.23673 0.43066,18.01831 v 0 c 6.36756,7.76532 7.85712,18.11251 3.91272,27.18018 -3.9444,9.06767 -12.7301,15.49351 -23.07849,16.87953 -0.0729,8.51033 -5.05414,16.31931 -13.02356,20.41701 -7.96942,4.09772 -17.68268,3.84427 -25.39587,-0.66263 -3.2854,10.1926 -12.53272,17.6922 -23.74671,19.25873 -11.21405,1.56653 -22.38446,-3.08087 -28.68524,-11.93434 -7.72339,4.3639 -16.99084,5.621 -25.71185,3.48773 -8.72095,-2.13329 -16.16077,-7.47725 -20.64124,-14.8264 v 0 c -7.89233,0.86539 -15.52319,-2.96604 -19.10534,-9.59276 -3.58216,-6.62671 -2.35303,-14.63803 3.07733,-20.05798 v 0 c -7.04022,-3.8826 -10.63251,-11.58696 -8.90375,-19.09555 1.72882,-7.50861 8.38696,-13.11998 16.50256,-13.90799 z"
id="path4059"
inkscape:connector-curvature="0"
style="fill:#f6cd4c;fill-rule:evenodd" />
<path
d="m 597.06497,212.10764 v 0 c 3.32227,1.83222 7.16028,2.66335 10.99872,2.38181 m 5.02759,27.2696 c 1.65063,-0.18098 3.26861,-0.56427 4.81219,-1.13993 m 41.53858,12.47812 c -1.16095,-1.63134 -2.133,-3.37456 -2.89954,-5.19996 m 55.3335,-2.12474 v 0 c 0.59893,-1.85826 0.98706,-3.77081 1.15777,-5.70575 m 37.26019,-14.04791 c 0.0776,-9.06056 -5.41461,-17.3565 -14.11767,-21.32448 m 33.2832,-22.73425 c -1.40942,3.08528 -3.56109,5.8222 -6.28625,7.99617 m -8.91071,-37.53403 v 0 c 0.24011,1.24591 0.35126,2.51055 0.33185,3.77639 m -37.17371,-13.02565 v 0 c -1.31659,1.45164 -2.4013,3.07382 -3.22033,4.81604 m -28.81055,-1.97402 v 0 c -0.70166,1.31851 -1.22559,2.71372 -1.55975,4.15347 m -35.13446,1.13608 v 0 c 2.04858,1.12421 3.94384,2.47731 5.64404,4.02959 m -49.5199,23.36855 v 0 c 0.20844,1.4353 0.53784,2.85292 0.98499,4.23957"
id="path4061"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 636.30257,211.8428 h -0.1875 q -0.78125,0 -1.25,-0.48437 -0.46875,-0.48438 -0.46875,-1.17188 0,-0.46875 0.5,-2.92187 l 1.375,-7.10938 q 0.46875,-2.42187 2.0625,-13.4375 l 0.39063,-2.70312 q 0.14062,-1.01563 1.10937,-1.84375 0.96875,-0.84375 1.78125,-0.84375 0.4375,0 2.9375,1.3125 2.70313,1.45312 3.20313,1.59375 5.09375,1.875 8.375,5.6875 3.29687,3.8125 3.29687,8.23437 0,4.04688 -2.40625,7.92188 -2.39062,3.875 -5.78125,5.71875 -3.39062,1.84375 -8.67187,1.84375 -1.53125,0 -3.46875,-0.53125 -1.9375,-0.51563 -2.79688,-1.26563 z m 6,-25.40625 -1.64062,11.20313 -2,10.67187 q 0.0625,0.0469 0.14062,0.0781 1.84375,1.32812 4.375,1.32812 5.01563,0 7.375,-1.79687 2.35938,-1.79688 3.60938,-4.21875 1.25,-2.42188 1.25,-4.92188 0,-2.51562 -1.40625,-4.64062 -1.39063,-2.125 -3.76563,-3.57813 -2.35937,-1.46875 -7.9375,-4.125 z m 32.16406,24.28125 q -4.21875,2.03125 -6.01562,2.03125 -7.29688,0 -7.29688,-6.95312 0,-6.46875 4.07813,-10.46875 4.09375,-4 8.98437,-4 1.89063,0 3.90625,0.96875 2.03125,0.96875 2.03125,2.25 0,0.79687 -0.53125,1.25 -0.39062,0.76562 -1.10937,4.40625 -0.70313,3.64062 -0.70313,5.25 0,1.96875 1.01563,5.42187 l 0.0781,0.3125 q -0.4375,1.82813 -2.17187,1.82813 -0.29688,0 -1.09375,-0.79688 -0.79688,-0.79687 -1.17188,-1.5 z m 1.42188,-15.1875 q -1.21875,-0.73437 -1.95313,-0.73437 -3.25,0 -6.07812,2.95312 -2.82813,2.9375 -2.82813,7.3125 0,4.29688 3.71875,4.29688 2.90625,0 5.46875,-2.15625 0.3125,-8.1875 1.67188,-11.67188 z m 16.45312,-0.34375 q -0.23437,0.9375 -0.60937,2.1875 -1.64063,5.42188 -1.64063,8.26563 0,3.35937 1.5625,3.35937 1.54688,0 2.28125,-0.95312 1.01563,-1.32813 1.875,-1.32813 0.65625,0 1.25,0.48438 0.59375,0.48437 0.59375,1.26562 0,1.71875 -2.15625,2.95313 -2.15625,1.21875 -4.29687,1.21875 -5.03125,0 -5.03125,-6.28125 0,-3.4375 1.54687,-8.39063 0.40625,-1.3125 0.78125,-2.625 -4.25,-0.20312 -4.85937,-0.70312 -0.60938,-0.5 -0.60938,-1.5625 0,-0.85938 0.57813,-1.42188 0.59375,-0.57812 1.60937,-0.57812 l 4.03125,0.32812 q 0.125,-0.51562 0.1875,-1.17187 0.0312,-0.67188 0.125,-1.48438 l 0.34375,-2.42187 q 0.10938,-0.79688 0.71875,-1.34375 0.625,-0.54688 1.46875,-0.54688 1.64063,0 1.64063,2.625 0,0.76563 -0.23438,1.98438 l -0.39062,2.35937 q 2.60937,-0.32812 2.78125,-0.32812 2.03125,0 2.79687,0.40625 0.76563,0.39062 0.76563,1.46875 0,0.95312 -0.65625,1.54687 -0.65625,0.57813 -1.57813,0.57813 l -1.92187,-0.0781 q -1.45313,0 -2.95313,0.1875 z m 21.44141,15.53125 q -4.21875,2.03125 -6.01563,2.03125 -7.29687,0 -7.29687,-6.95312 0,-6.46875 4.07812,-10.46875 4.09375,-4 8.98438,-4 1.89062,0 3.90625,0.96875 2.03125,0.96875 2.03125,2.25 0,0.79687 -0.53125,1.25 -0.39063,0.76562 -1.10938,4.40625 -0.70312,3.64062 -0.70312,5.25 0,1.96875 1.01562,5.42187 l 0.0781,0.3125 q -0.4375,1.82813 -2.17188,1.82813 -0.29687,0 -1.09375,-0.79688 -0.79687,-0.79687 -1.17187,-1.5 z m 1.42187,-15.1875 q -1.21875,-0.73437 -1.95312,-0.73437 -3.25,0 -6.07813,2.95312 -2.82812,2.9375 -2.82812,7.3125 0,4.29688 3.71875,4.29688 2.90625,0 5.46875,-2.15625 0.3125,-8.1875 1.67187,-11.67188 z"
id="path4063"
inkscape:connector-curvature="0"
style="fill:#000000;fill-rule:nonzero" />
<path
d="m 604.81711,394.03214 v 0 c -1.51233,-10.41397 3.45307,-20.72311 12.78925,-26.55298 9.33618,-5.82983 21.40576,-6.1579 31.08722,-0.845 v 0 c 3.42944,-6.05511 9.70636,-10.23575 16.93207,-11.27734 7.2257,-1.0416 14.55151,1.17819 19.76159,5.98788 v 0 c 2.92145,-5.48993 8.65778,-9.17847 15.1734,-9.75674 6.51569,-0.57825 12.88855,2.03564 16.85718,6.91409 v 0 c 5.27802,-5.81934 13.67554,-8.2695 21.55884,-6.29031 7.8833,1.97915 13.83649,8.03222 15.28363,15.53994 v 0 c 6.46649,1.65271 11.8529,5.85413 14.76764,11.51874 2.91473,5.66461 3.07178,12.23673 0.43066,18.01831 v 0 c 6.36756,7.76532 7.85712,18.11249 3.91272,27.18018 -3.9444,9.06766 -12.7301,15.4935 -23.07849,16.87952 -0.0729,8.51034 -5.05414,16.3193 -13.02356,20.41702 -7.96942,4.09769 -17.68268,3.84427 -25.39587,-0.66266 -3.2854,10.19263 -12.53272,17.69223 -23.74671,19.25876 -11.21405,1.56653 -22.38446,-3.08087 -28.68524,-11.93436 -7.72339,4.36392 -16.99084,5.621 -25.71185,3.48773 -8.72095,-2.13327 -16.16077,-7.47723 -20.64124,-14.82638 v 0 c -7.89233,0.86539 -15.52319,-2.96607 -19.10534,-9.59277 -3.58216,-6.62671 -2.35303,-14.63804 3.07733,-20.05799 v 0 c -7.04022,-3.8826 -10.63257,-11.58694 -8.90375,-19.09555 1.72882,-7.5086 8.38696,-13.11996 16.50256,-13.90799 z"
id="path4065"
inkscape:connector-curvature="0"
style="fill:#f6cd4c;fill-rule:evenodd" />
<path
d="m 597.06237,427.43705 v 0 c 3.32227,1.83218 7.16028,2.66333 10.99872,2.38181 m 5.02759,27.26959 c 1.65063,-0.181 3.26861,-0.56427 4.81219,-1.13992 m 41.53858,12.47811 c -1.16095,-1.63134 -2.133,-3.37457 -2.89954,-5.19998 m 55.3335,-2.12472 v 0 c 0.59893,-1.85825 0.98706,-3.77082 1.15777,-5.70575 m 37.26019,-14.04792 c 0.0776,-9.06054 -5.41461,-17.3565 -14.11767,-21.32449 m 33.2832,-22.73425 c -1.40942,3.0853 -3.56109,5.82223 -6.28625,7.99618 m -8.91071,-37.53402 v 0 c 0.24011,1.24591 0.35126,2.51056 0.33185,3.77639 m -37.17371,-13.02566 v 0 c -1.31659,1.45163 -2.4013,3.07382 -3.22033,4.81604 m -28.81055,-1.974 v 0 c -0.70166,1.31848 -1.22559,2.71372 -1.55975,4.15347 m -35.13446,1.13608 v 0 c 2.04858,1.12421 3.94384,2.4773 5.64404,4.02957 m -49.5199,23.36856 v 0 c 0.20844,1.43531 0.53784,2.85291 0.98499,4.23957"
id="path4067"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 636.29997,427.17219 h -0.1875 q -0.78125,0 -1.25,-0.48437 -0.46875,-0.48438 -0.46875,-1.17188 0,-0.46875 0.5,-2.92187 l 1.375,-7.10938 q 0.46875,-2.42187 2.0625,-13.4375 l 0.39063,-2.70312 q 0.14062,-1.01563 1.10937,-1.84375 0.96875,-0.84375 1.78125,-0.84375 0.4375,0 2.9375,1.3125 2.70313,1.45312 3.20313,1.59375 5.09375,1.875 8.375,5.6875 3.29687,3.8125 3.29687,8.23437 0,4.04688 -2.40625,7.92188 -2.39062,3.875 -5.78125,5.71875 -3.39062,1.84375 -8.67187,1.84375 -1.53125,0 -3.46875,-0.53125 -1.9375,-0.51563 -2.79688,-1.26563 z m 6,-25.40625 -1.64062,11.20313 -2,10.67187 q 0.0625,0.0469 0.14062,0.0781 1.84375,1.32812 4.375,1.32812 5.01563,0 7.375,-1.79687 2.35938,-1.79688 3.60938,-4.21875 1.25,-2.42188 1.25,-4.92188 0,-2.51562 -1.40625,-4.64062 -1.39063,-2.125 -3.76563,-3.57813 -2.35937,-1.46875 -7.9375,-4.125 z m 32.16406,24.28125 q -4.21875,2.03125 -6.01562,2.03125 -7.29688,0 -7.29688,-6.95312 0,-6.46875 4.07813,-10.46875 4.09375,-4 8.98437,-4 1.89063,0 3.90625,0.96875 2.03125,0.96875 2.03125,2.25 0,0.79687 -0.53125,1.25 -0.39062,0.76562 -1.10937,4.40625 -0.70313,3.64062 -0.70313,5.25 0,1.96875 1.01563,5.42187 l 0.0781,0.3125 q -0.4375,1.82813 -2.17187,1.82813 -0.29688,0 -1.09375,-0.79688 -0.79688,-0.79687 -1.17188,-1.5 z m 1.42188,-15.1875 q -1.21875,-0.73437 -1.95313,-0.73437 -3.25,0 -6.07812,2.95312 -2.82813,2.9375 -2.82813,7.3125 0,4.29688 3.71875,4.29688 2.90625,0 5.46875,-2.15625 0.3125,-8.1875 1.67188,-11.67188 z m 16.45312,-0.34375 q -0.23437,0.9375 -0.60937,2.1875 -1.64063,5.42188 -1.64063,8.26563 0,3.35937 1.5625,3.35937 1.54688,0 2.28125,-0.95312 1.01563,-1.32813 1.875,-1.32813 0.65625,0 1.25,0.48438 0.59375,0.48437 0.59375,1.26562 0,1.71875 -2.15625,2.95313 -2.15625,1.21875 -4.29687,1.21875 -5.03125,0 -5.03125,-6.28125 0,-3.4375 1.54687,-8.39063 0.40625,-1.3125 0.78125,-2.625 -4.25,-0.20312 -4.85937,-0.70312 -0.60938,-0.5 -0.60938,-1.5625 0,-0.85938 0.57813,-1.42188 0.59375,-0.57812 1.60937,-0.57812 l 4.03125,0.32812 q 0.125,-0.51562 0.1875,-1.17187 0.0312,-0.67188 0.125,-1.48438 l 0.34375,-2.42187 q 0.10938,-0.79688 0.71875,-1.34375 0.625,-0.54688 1.46875,-0.54688 1.64063,0 1.64063,2.625 0,0.76563 -0.23438,1.98438 l -0.39062,2.35937 q 2.60937,-0.32812 2.78125,-0.32812 2.03125,0 2.79687,0.40625 0.76563,0.39062 0.76563,1.46875 0,0.95312 -0.65625,1.54687 -0.65625,0.57813 -1.57813,0.57813 l -1.92187,-0.0781 q -1.45313,0 -2.95313,0.1875 z m 21.44141,15.53125 q -4.21875,2.03125 -6.01563,2.03125 -7.29687,0 -7.29687,-6.95312 0,-6.46875 4.07812,-10.46875 4.09375,-4 8.98438,-4 1.89062,0 3.90625,0.96875 2.03125,0.96875 2.03125,2.25 0,0.79687 -0.53125,1.25 -0.39063,0.76562 -1.10938,4.40625 -0.70312,3.64062 -0.70312,5.25 0,1.96875 1.01562,5.42187 l 0.0781,0.3125 q -0.4375,1.82813 -2.17188,1.82813 -0.29687,0 -1.09375,-0.79688 -0.79687,-0.79687 -1.17187,-1.5 z m 1.42187,-15.1875 q -1.21875,-0.73437 -1.95312,-0.73437 -3.25,0 -6.07813,2.95312 -2.82812,2.9375 -2.82812,7.3125 0,4.29688 3.71875,4.29688 2.90625,0 5.46875,-2.15625 0.3125,-8.1875 1.67187,-11.67188 z"
id="path4069"
inkscape:connector-curvature="0"
style="fill:#000000;fill-rule:nonzero" />
<path
d="m 299.38846,189.04987 h 70.36065 v -6.68504 l 8.40525,13.37008 -8.40525,13.37007 v -6.68502 h -70.36065 z"
id="path4071"
inkscape:connector-curvature="0"
style="fill:#7c7ce0;fill-rule:evenodd;stroke-width:0.79288208" />
<path
d="M 427.88773,353.66272 H 553.16596 V 468.46329 H 427.88773 Z"
id="path4075"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<g
transform="matrix(0.45555722,0,0,0.45555774,379.83689,353.66273)"
id="g4082">
<clipPath
id="p.3">
<path
d="M 0,0 H 275 V 252 H 0 Z"
id="path4077"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
<image
clip-path="url(#p.3)"
width="275"
height="252"
x="0"
y="0"
preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/magic.gif"
id="image4080"
style="fill:#000000" />
</g>
<path
d="m 660.00147,264.20093 21.66931,-21.66928 21.66932,21.66928 h -10.83466 v 85.7638 h 10.83466 l -21.66932,21.66928 -21.66931,-21.66928 h 10.83466 v -85.7638 z"
id="path4088"
inkscape:connector-curvature="0"
style="fill:#db4437;fill-rule:evenodd" />
<path
d="m 660.00147,264.20093 21.66931,-21.66928 21.66932,21.66928 h -10.83466 v 85.7638 h 10.83466 l -21.66932,21.66928 -21.66931,-21.66928 h 10.83466 v -85.7638 z"
id="path4090"
inkscape:connector-curvature="0"
style="fill-rule:evenodd;stroke:#666666;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
<path
d="m 554.53672,285.04017 c 42.66144,-12.09317 85.32288,12.0932 127.98438,0 42.66143,-12.09317 85.32287,12.0932 127.98431,0 l -1.70062,50.79135 c -42.66144,12.09317 -85.32288,-12.09317 -127.98432,0 -42.66143,12.09317 -85.32294,-12.09317 -127.98437,0 z"
id="path4092"
inkscape:connector-curvature="0"
style="fill:#db4437;fill-rule:evenodd" />
<path
d="m 587.72404,321.45273 q -0.84375,-0.5 -1.26562,-1.0625 -0.40625,-0.5625 -0.40625,-1.07812 0,-0.67188 0.67187,-1.07813 0.125,-0.0625 0.35938,-0.0625 0.3125,0 0.67187,0.10938 0.375,0.10937 0.57813,0.29687 0.82812,0.76563 1.98437,0.76563 0.5,0 1.46875,-0.21875 1.25,-0.39063 1.96875,-0.95313 0.71875,-0.57812 1.70313,-1.79687 0.60937,-0.79688 0.90625,-1.60938 0.3125,-0.82812 0.3125,-1.5 0,-0.73437 -0.34375,-1.15625 -0.64063,-0.85937 -1.4375,-1.28125 -0.78125,-0.4375 -1.82813,-0.4375 -0.9375,0 -1.82812,0.25 l -1.85938,0.39063 q -0.125,0.0312 -0.34375,0.0469 -0.20312,0.0156 -0.59375,0.95313 -0.375,0.92187 -1.29687,3.51562 -0.67188,1.875 -1.125,2.70313 -0.45313,0.8125 -0.79688,0.8125 -0.39062,0 -0.82812,-0.70313 -0.0937,-0.15625 -0.40625,-0.64062 -0.29688,-0.48438 -0.29688,-0.9375 0,-0.21875 0.15625,-0.53125 0.15625,-0.29688 0.26563,-0.57813 0.10937,-0.29687 0.25,-0.79687 0.40625,-1.25 2.40625,-6.20313 2,-4.96875 3.125,-7.5 0.79687,-1.82812 0.48437,-1.92187 0,-0.0937 -0.10937,-0.28125 -0.0937,-0.20313 -0.0625,-0.29688 0,-0.1875 1.14062,-0.5625 1.14063,-0.39062 1.54688,-0.32812 h 0.20312 q 0.40625,0 0.40625,-0.15625 0.0625,0 0.59375,-0.0625 0.53125,-0.0625 1.14063,0 2.26562,0.1875 3.57812,1.29687 1.3125,1.09375 1.28125,3.23438 -0.0625,0.84375 -0.92187,2.28125 -0.85938,1.4375 -1.89063,2.35937 -0.51562,0.45313 -1.17187,0.95313 -0.65625,0.5 -0.84375,0.625 l 0.79687,0.28125 q 1.5,0.57812 2.32813,1.42187 0.84375,0.82813 1.1875,2.26563 0.0781,0.3125 0.0781,0.57812 0,0.70313 -0.25,1.59375 -0.23437,0.89063 -0.53125,1.5625 -0.1875,0.20313 -0.3125,0.42188 -0.125,0.21875 -0.15625,0.28125 -0.0312,0.14062 -0.57812,0.98437 -0.54688,0.84375 -0.96875,1.35938 -2.10938,1.79687 -3.48438,2.29687 -1.34375,0.60938 -3.20312,0.60938 -1.65625,0 -2.45313,-0.51563 z m 2.29688,-13.26562 q 0.32812,-0.0781 1.28125,-0.28125 0.95312,-0.20313 1.375,-0.35938 0.0625,-0.0312 0.39062,-0.20312 0.34375,-0.1875 0.70313,-0.54688 2.01562,-1.875 2.8125,-3.39062 0.79687,-1.3125 0.79687,-1.84375 0,-0.39063 -0.34375,-0.75 -0.60937,-0.70313 -2.01562,-0.70313 -0.60938,0 -1.03125,0.14063 -0.45313,0.125 -0.65625,0.29687 -0.20313,0.17188 -0.20313,0.53125 0,0.25 -0.17187,0.5625 -0.15625,0.29688 -0.34375,0.48438 -0.0312,0.20312 -0.17188,0.39062 -0.14062,0.1875 -0.21875,0.25 0,0.0781 -0.4375,0.875 -0.42187,0.79688 -0.79687,1.78125 -0.96875,2.28125 -0.96875,2.76563 z m 11.90643,10.29687 q -0.0937,-0.1875 -0.54688,-0.5 -0.45312,-0.32812 -0.45312,-0.39062 0,-0.14063 -0.14063,-0.375 -0.14062,-0.23438 -0.26562,-0.4375 -0.26563,-0.25 0.34375,-1.65625 0.60937,-1.40625 1.64062,-3.32813 0.1875,-0.1875 0.48438,-0.67187 0.3125,-0.48438 0.40625,-0.54688 0,-0.125 0.0937,-0.23437 0.0937,-0.10938 0.23437,-0.10938 l 0.25,-0.51562 q 0.125,-0.29688 1.25,-1.375 1.125,-1.09375 1.76563,-1.46875 0.59375,-0.42188 1.3125,-0.42188 0.76562,0 1.29687,0.45313 l 0.875,0.73437 q 0.95313,0.79688 1.07813,0.92188 l 1.1875,1.0625 -0.54688,1.59375 q -0.28125,0.85937 -0.28125,1.64062 0,1.20313 0.67188,2.26563 0.15625,0.34375 0.42187,0.54687 0.28125,0.1875 0.73438,0.25 0.70312,0.125 0.70312,0.70313 0,0.70312 -0.26562,1.0625 -0.25,0.34375 -0.60938,0.4375 -0.5,0.0781 -0.70312,0.0781 -0.92188,0 -1.6875,-0.48437 -0.76563,-0.48438 -1.34375,-1.64063 -0.125,-0.1875 -0.34375,-0.6875 -0.20313,-0.51562 -0.23438,-0.71875 -0.21875,0.32813 -0.73437,0.84375 -3.29688,3.32813 -5.34375,3.32813 -0.70313,0 -1.25,-0.35938 z m 1.76562,-2.65625 q 0.5,0.0625 1.98438,-1.15625 0.15625,-0.125 0.34375,-0.29687 0.1875,-0.17188 0.42187,-0.375 0.89063,-0.73438 1.39063,-1.21875 0.5,-0.5 1,-1.20313 l 0.70312,-1.15625 q -0.3125,-0.70312 -0.57812,-0.9375 -0.25,-0.25 -0.5,-0.25 -0.64063,0 -1.60938,1.14063 -0.95312,1.14062 -2.29687,3.4375 -0.35938,0.57812 -0.57813,1.20312 -0.21875,0.625 -0.28125,0.8125 z m 14.94391,3.8125 q -0.78125,0 -1.59375,-0.32812 -0.8125,-0.3125 -1.375,-0.78125 -0.5625,-0.46875 -0.5625,-0.84375 0,-0.39063 0.35938,-0.48438 l 0.15625,-0.125 q 0.0625,-0.0312 0.25,-0.0312 0.14062,-0.0312 0.32812,0.0156 0.1875,0.0469 0.3125,0.0469 0.23438,0.0625 0.67188,0.125 l 0.84375,0.0937 q 0.64062,0 1.1875,-0.125 0.5625,-0.125 0.5625,-0.28125 0,-0.10937 -0.25,-0.32812 -0.25,-0.21875 -0.54688,-0.21875 0,-0.125 -0.98437,-0.85938 -0.54688,-0.42187 -1.04688,-0.82812 -0.5,-0.42188 -0.5,-0.625 l 0.0312,-0.0625 v -0.15625 q -0.21875,0 -0.42187,-0.60938 -0.1875,-0.60937 -0.1875,-1.1875 0,-1.07812 0.95312,-2.39062 0.96875,-1.3125 2.39063,-2.35938 1.4375,-1.0625 2.71875,-1.45312 0.46875,-0.21875 1.40625,-0.21875 0.60937,0 0.82812,0.125 l 0.25,-0.0625 q 0.39063,0 1,0.64062 0.60938,0.64063 0.76563,1.01563 0,1.0625 -0.0625,1.70312 -0.0625,0.64063 -0.32813,0.64063 -0.0312,0 -0.0937,0.0312 -0.0625,0.0312 -0.0625,0.15625 0,0.45312 -0.78125,1 -0.78125,0.53125 -1.42187,0.53125 l -0.125,-0.0312 q -0.20313,-0.0312 -0.57813,-0.375 -0.35937,-0.35938 -0.35937,-0.70313 0,-0.0937 0.375,-0.51562 0.39062,-0.42188 0.65625,-0.8125 0.28125,-0.40625 0.28125,-0.85938 0,-0.15625 -0.10938,-0.20312 -0.10937,-0.0469 -0.4375,-0.0469 -1.75,0.79688 -2.51562,1.5 -0.10938,0.20313 -0.1875,0.29688 -0.0781,0.0937 -0.20313,0.0937 0,0 0,-0.0156 0,-0.0156 -0.0312,-0.0156 -0.0625,0 -0.42187,0.375 -0.34375,0.35937 -0.65625,0.84375 -0.29688,0.46875 -0.35938,0.89062 l -0.0625,0.0937 v 0.15625 q -0.0312,0.0625 -0.0312,0.26563 0,0.28125 0.25,0.59375 0.25,0.29687 1.03125,0.84375 1.4375,1.09375 1.95313,1.625 0.53125,0.51562 0.67187,0.9375 0.28125,0.57812 0.28125,1.01562 0,1.09375 -1.14062,1.6875 -1.125,0.59375 -3.07813,0.59375 z m 13.21564,-13.79687 q -0.10937,0.0937 -0.20312,0.0937 -0.125,0 -0.40625,-0.1875 -0.32813,-0.0312 -0.85938,-0.54688 -0.51562,-0.51562 -0.51562,-0.67187 -0.0312,-0.39063 0.0625,-1.10938 0.0937,-0.71875 0.35937,-1.28125 0.28125,-0.57812 0.71875,-0.60937 0.23438,0.0312 0.60938,0.14062 0.39062,0.10938 0.875,0.29688 0.59375,1.21875 0.59375,2.07812 0,1.28125 -1.23438,1.79688 z m -2.95312,13.28125 q -0.76563,0 -1.32813,-0.79688 -0.5625,-0.79687 -0.6875,-1.98437 0.125,-1.1875 0.1875,-1.73438 0.35938,-1.5625 0.89063,-3.6875 0.53125,-2.14062 0.625,-2.35937 0.15625,-0.28125 0.5,-0.28125 0.35937,0 0.78125,0.26562 0.4375,0.26563 0.73437,0.75 0.25,0.35938 0.25,0.79688 0,0.67187 -0.64062,2.82812 -0.4375,1.59375 -0.57813,2.26563 -0.125,0.67187 -0.0937,1.4375 -0.125,1.4375 -0.25,1.98437 -0.125,0.54688 -0.39062,0.51563 z m 6.8189,-0.21875 q -0.95312,0 -1.25,-0.10938 -0.28125,-0.125 -0.70312,-0.5 -0.0937,-0.23437 -0.40625,-0.60937 -0.29688,-0.39063 -0.48438,-0.57813 -0.0937,-0.35937 -0.0937,-0.82812 0,-0.84375 0.23438,-1.95313 0.23437,-1.125 0.59375,-2.04687 0.125,-0.26563 0.20312,-0.48438 0.0781,-0.21875 0.10938,-0.28125 0,-0.23437 0.8125,-1.39062 0.82812,-1.17188 1.3125,-1.625 0.21875,-0.25 0.67187,-0.53125 0.46875,-0.29688 0.82813,-0.42188 0.0937,-0.0312 0.78125,-0.28125 0.6875,-0.26562 1.26562,-0.26562 0.51563,0 1,0.20312 0.5,0.20313 0.82813,0.65625 0.82812,0.82813 0.82812,2.17188 0,0.26562 -0.0625,0.57812 -0.17187,0.64063 -0.67187,1.04688 -0.48438,0.39062 -1.03125,0.39062 -0.54688,0 -0.92188,-0.40625 -0.125,-0.32812 -0.125,-0.42187 0,-0.0937 0.125,-0.54688 0.125,-0.57812 0.125,-0.73437 0,-0.40625 -0.28125,-0.40625 -0.39062,0 -0.625,0.1875 -0.21875,0.1875 -0.82812,0.82812 -0.40625,0.45313 -0.96875,1.29688 -0.5625,0.84375 -0.875,1.60937 -0.29688,0.875 -0.40625,1.25 -0.10938,0.35938 -0.10938,0.42188 0,0.125 -0.0312,0.23437 -0.0312,0.10938 -0.0312,0.25 0,0.21875 0.0937,0.34375 0.0937,0.125 0.34375,0.26563 0.0781,0.0312 0.23438,0.125 0.15625,0.0937 0.3125,0.0937 0.20312,0 0.54687,-0.15625 0.35938,-0.17188 0.45313,-0.20313 0.40625,-0.1875 1.17187,-0.82812 l 0.78125,-0.57813 q 0.21875,-0.15625 0.39063,-0.26562 0.1875,-0.10938 0.28125,-0.15625 0.0937,-0.0469 0.0937,0.0156 0,0.125 0.0937,0.125 0.0937,0 0.21875,-0.125 0.125,-0.20312 0.45312,-0.20312 0.25,0 0.51563,0.15625 0.28125,0.14062 0.28125,0.23437 l -0.0312,0.15625 q 0,0.15625 0.15625,0.32813 0,0.125 -0.0781,0.32812 -0.0781,0.20313 -0.20312,0.34375 -0.32813,0.46875 -0.73438,0.98438 -0.39062,0.51562 -0.54687,0.70312 h -0.0312 q -0.15625,0 -0.45313,0.29688 -0.46875,0.4375 -0.82812,0.4375 -0.125,0.0625 -0.29688,0.15625 -0.15625,0.0781 -0.15625,0.17187 -0.53125,0.54688 -2.875,0.54688 z m 10.59815,-0.32813 q -0.54688,0.125 -1.3125,-0.57812 -0.76563,-0.70313 -0.90625,-1.3125 -0.0625,-0.3125 -0.0625,-0.54688 0,-1.23437 0.79687,-3.01562 0.8125,-1.78125 2.17188,-3.34375 1.35937,-1.5625 2.98437,-2.26563 0.67188,-0.35937 0.85938,-0.35937 0.20312,0 0.875,0.35937 0.64062,0.15625 1.01562,0.34375 0.39063,0.1875 0.70313,0.60938 0.42187,0.82812 0.60937,2.04687 0.20313,0.67188 0.20313,0.95313 0,0.26562 -0.20313,0.45312 -0.0312,0.0312 -0.10937,0.125 -0.0781,0.0937 -0.0781,0.15625 0,0.0469 0.0625,0.0469 0,0.0312 -0.0312,0.14062 -0.0312,0.10938 -0.125,0.29688 -0.39062,1.21875 -0.39062,1.89062 0,0.21875 0.0625,0.54688 0.32812,0.82812 0.8125,0.82812 0.4375,0 1.01562,-0.57812 0.48438,-0.3125 0.96875,-0.3125 0.28125,0 0.28125,0.15625 0.0312,0.0312 0.125,0.0937 0.0937,0.0625 0.1875,0.0625 0.14063,0 0.14063,0.29687 0,0.25 -0.125,0.67188 -0.10938,0.40625 -0.26563,0.67187 -0.21875,0.21875 -0.35937,0.39063 -0.125,0.15625 -0.125,0.25 l -0.46875,0.4375 q -0.60938,0.35937 -1.45313,0.35937 -0.85937,0 -1.375,-0.35937 -0.5,-0.34375 -0.95312,-0.875 -0.45313,-0.53125 -0.57813,-0.9375 l -0.15625,-0.32813 -0.9375,0.60938 -0.79687,0.82812 q -0.25,0.32813 -0.4375,0.32813 h -0.32813 q 0,0.21875 -0.90625,0.54687 -0.89062,0.3125 -1.39062,0.3125 z m 0.67187,-2.6875 q 0.34375,0 1.4375,-0.8125 1.10938,-0.8125 1.5,-1.29687 0.125,-0.15625 0.32813,-0.45313 0.21875,-0.3125 0.375,-0.46875 0.51562,-0.60937 0.75,-0.96875 0.25,-0.375 0.25,-0.73437 0,-0.25 -0.23438,-0.70313 -0.125,-0.39062 -0.375,-0.39062 -0.14062,0 -0.73437,0.23437 -0.60938,0.34375 -1.4375,1.4375 -0.8125,1.07813 -1.42188,2.20313 -0.60937,1.125 -0.60937,1.64062 0,0.0937 0.0625,0.1875 0.0781,0.0937 0.10937,0.125 z m 11.06427,4.21875 q -0.48437,-0.15625 -0.71875,-0.57812 -0.23437,-0.40625 -0.23437,-1.20313 0,-1.3125 0.25,-2.5625 l 0.0312,-0.25 q 0.0312,-0.20312 0.0937,-0.625 0.0625,-0.4375 0.0937,-0.25 l 0.20312,-0.85937 q 0.21875,-0.89063 0.42188,-1.65625 0.21875,-0.78125 0.34375,-1.125 l 0.34375,-1.125 q 0.96875,-3.10938 1.79687,-4.1875 0.0312,-0.125 0.0937,-0.23438 0.0625,-0.125 0.0937,-0.25 0,-0.51562 0.20313,-0.79687 0.28125,-0.48438 0.46875,-0.85938 l 0.14062,-0.35937 0.89063,-1.82813 q 0.0937,-0.3125 0.39062,-0.6875 0.3125,-0.39062 0.625,-0.53125 l -0.0312,-0.0312 q 0.17188,-0.25 0.59375,-0.4375 0.4375,-0.20312 0.75,-0.20312 0.42188,0 0.67188,0.26562 0.26562,0.25 0.26562,0.70313 v 0.3125 q 0,0.29687 -0.0469,0.45312 -0.0469,0.15625 -0.21875,0.39063 h 0.0312 q 0.0469,0.0312 -0.0469,0.20312 -0.0781,0.17188 -0.14062,0.32813 -0.21875,0.57812 -0.3125,0.64062 -0.23438,0.29688 -1.48438,2.71875 -1.23437,2.40625 -1.59375,3.32813 l -0.125,0.48437 -0.0625,0.20313 q -0.35937,0.76562 -0.39062,0.82812 l -0.51563,1.98438 -0.125,0.67187 -0.28125,0.73438 -0.1875,0.60937 q -0.0781,0.15625 -0.125,0.42188 -0.0469,0.25 -0.10937,0.3125 l -0.0312,0.15625 q -0.0312,0.125 -0.21875,0.73437 -0.29688,0.90625 -0.42188,1.5 -0.125,0.57813 -0.15625,1.25 l -0.0312,0.14063 q -0.0937,0.70312 -0.40625,1.04687 -0.29687,0.35938 -0.78125,0.21875 z m 6.82099,-0.59375 q -0.0937,0.0312 -0.26563,0.0312 -0.67187,0 -1.125,-0.76562 -0.4375,-0.78125 -0.4375,-1.96875 0,-0.625 0.125,-1.23438 0.35938,-0.9375 0.35938,-1.34375 0.0937,-0.60937 0.65625,-2.1875 0.5625,-1.59375 0.71875,-1.92187 0.125,0 0.21875,-0.20313 0.0937,-0.20312 0.0937,-0.46875 0,-0.28125 0.10937,-0.57812 0.125,-0.3125 0.25,-0.4375 0,-0.0937 0.0781,-0.34375 0.0781,-0.26563 0.0781,-0.35938 0,-0.0625 0.0781,-0.23437 0.0781,-0.1875 0.0781,-0.28125 0.51563,-0.98438 1.79688,-4.45313 1.28125,-3.48437 1.28125,-3.85937 0,-0.26563 0.1875,-0.48438 0.20312,-0.21875 0.45312,-0.21875 0.0937,-0.10937 0.40625,-0.15625 0.32813,-0.0469 0.51563,-0.0469 l 0.51562,0.51562 q 0.35938,0.32813 0.54688,0.57813 0.1875,0.25 0.1875,0.60937 0,0.57813 -0.20313,1.14063 -0.20312,0.54687 -1.20312,2.625 -0.0625,0.17187 -0.51563,1.15625 -0.3125,0.64062 -0.3125,0.73437 0,0.0937 -0.14062,0.40625 -0.125,0.29688 -0.21875,0.625 -0.25,0.51563 -0.8125,1.8125 -0.5625,1.29688 -0.5625,1.42188 0,0.34375 -0.15625,0.5 -0.0937,0 -0.125,0.15625 -0.0312,0.14062 -0.0312,0.20312 0,0.3125 -0.51563,1.70313 l -0.28125,0.92187 q -0.60937,2.07813 -0.85937,3.04688 -0.23438,0.95312 -0.23438,1.625 0,0.70312 -0.23437,1.15625 -0.21875,0.4375 -0.5,0.57812 z m 13.97729,-12.23437 q 0.29688,0.57812 0.29688,1.3125 0,0.9375 -0.42188,1.53125 -0.3125,0.45312 -0.5625,1.35937 -0.23437,0.89063 -0.45312,2.39063 -0.29688,1.21875 -0.71875,3.75 -0.5,1.34375 -0.625,2.01562 -0.0469,0.1875 -0.3125,0.79688 -0.26563,0.60937 -0.4375,0.95312 -1.4375,2.92188 -3.40625,4.39063 -1.96875,1.46875 -4.01563,1.59375 -0.51562,0.0625 -0.60937,0.0625 -0.28125,0 -0.53125,-0.0937 -0.23438,-0.0937 -0.51563,-0.25 -0.64062,-0.45313 -0.73437,-0.64063 -0.17188,-0.32812 -0.17188,-0.64062 0,-0.48438 0.45313,-0.65625 0.45312,-0.17188 1.03125,-0.17188 0.21875,0.125 0.76562,0.125 0.54688,0 0.64063,-0.125 0.0625,-0.0625 0.375,-0.21875 0.32812,-0.14062 0.73437,-0.23437 1.0625,-0.51563 1.625,-1.03125 0.5625,-0.5 1.23438,-1.625 0.79687,-1.21875 1.04687,-2.20313 0.42188,-1.0625 1,-3.14062 l -0.0312,0.0312 q -0.17187,0.1875 -0.8125,0.5625 -0.625,0.375 -0.76562,0.375 -0.0312,0 -0.0937,0.0312 -0.0625,0.0312 -0.0625,0.125 0,0.125 -0.82813,0.40625 -0.82812,0.26562 -1.0625,0.26562 -0.3125,0 -0.6875,-0.34375 -0.35937,-0.35937 -0.53125,-0.65625 -0.125,-0.28125 -0.46875,-0.57812 -0.39062,-0.375 -0.57812,-0.67188 -0.1875,-0.3125 -0.1875,-0.73437 0,-0.21875 0.0312,-0.34375 0.375,-1.4375 1.40625,-4.42188 1.04687,-3 1.53125,-3.73437 0.125,-0.20313 0.46875,-0.32813 0.35937,-0.125 0.71875,-0.125 0.64062,0 0.85937,0.375 0.39063,0.54688 0.39063,0.9375 0,0.4375 -0.45313,1.3125 -0.125,0.34375 -0.32812,0.76563 -0.1875,0.42187 -0.1875,0.51562 -0.0312,0.125 -0.1875,0.78125 -0.15625,0.65625 -0.28125,0.65625 -0.20313,0.1875 -0.4375,1.32813 -0.23438,1.14062 -0.23438,1.8125 0,0.375 0.0937,0.375 0.85938,0 1.65625,-0.45313 0.79688,-0.46875 1.67188,-1.5 0.125,-0.79687 0.35937,-1.71875 0.25,-0.9375 0.34375,-1.25 0.57813,-1.98437 0.76563,-3.20312 0.1875,-0.45313 0.28125,-0.53125 0.0937,-0.0781 0.42187,-0.0781 0.39063,0 0.84375,0.23438 0.46875,0.21875 0.6875,0.53125 z m 11.99976,13.79687 q -1.25,-0.0312 -1.78125,-0.59375 -0.51563,-0.5625 -0.51563,-1.57812 0,-0.26563 0.0625,-0.84375 0.125,-0.4375 0.1875,-0.89063 0.15625,-0.70312 0.32813,-0.82812 0.0312,-0.0312 0.0625,-0.125 0.0312,-0.10938 -0.0625,-0.17188 0,-0.40625 0.28125,-1.14062 0.0625,-0.23438 0.15625,-0.45313 0.0937,-0.21875 0.14062,-0.28125 -0.0781,-0.0781 -0.0781,-0.20312 0,-0.15625 0.17187,-0.3125 0.1875,-0.15625 0.26563,-0.28125 0.0781,-0.14063 -0.0156,-0.26563 -0.0625,-0.15625 0.60938,-1.4375 0.42187,-0.82812 0.45312,-0.9375 -0.0312,-0.0937 0.45313,-1.17187 0.5,-1.09375 0.79687,-1.64063 0.28125,-0.15625 -0.23437,-0.21875 -0.51563,-0.0625 -1.25,-0.0312 -1.04688,-0.0312 -1.625,-0.10937 -0.57813,-0.0937 -0.9375,-0.28125 -0.21875,-0.21875 -0.21875,-0.57813 0,-0.40625 0.29687,-0.75 0.3125,-0.35937 0.76563,-0.39062 0.82812,-0.0312 2.5625,-0.23438 l 1.625,-0.125 0.0937,-0.54687 q 0.15625,-0.28125 0.29687,-0.48438 0.15625,-0.21875 0.20313,-0.375 0.0469,-0.15625 0.0781,-0.48437 0.0937,-0.125 0.32813,-0.625 0.25,-0.51563 0.3125,-1 0.35937,-0.60938 0.65625,-1.17188 0.29687,-0.5625 0.375,-0.65625 0.5625,-0.85937 1.26562,-0.85937 0.39063,0 0.8125,0.25 0.40625,0.29687 0.625,0.51562 0.23438,0.21875 0.23438,0.60938 0,0.28125 -0.29688,0.82812 -0.125,0.25 -0.375,1.03125 -0.21875,0.28125 -0.48437,0.78125 -0.25,0.5 -0.34375,0.9375 -0.0625,0.15625 -0.20313,0.4375 -0.125,0.26563 -0.17187,0.3125 -0.0469,0.0469 -0.14063,0.0469 0.0312,0.20312 0.125,0.26562 0.0937,0.0625 0.32813,0.0625 0.15625,0 0.39062,-0.0312 0.25,-0.0312 0.59375,-0.0312 0.32813,-0.0312 0.5625,-0.0937 0.23438,-0.0625 0.46875,-0.0625 0.125,0 0.375,0.0937 0.15625,0.1875 0.15625,0.73438 0,0.40625 -0.125,0.78125 -0.125,0.375 -0.3125,0.46875 -0.15625,0.0312 -0.70312,0.1875 -0.54688,0.15625 -1.1875,0.1875 l -1.70313,0.15625 -0.53125,1.09375 q -0.23437,0.39062 -0.39062,0.75 -0.15625,0.34375 -0.21875,0.40625 0,0.15625 -0.26563,0.64062 -0.25,0.48438 -0.375,0.76563 -0.17187,0.45312 -0.39062,0.89062 -0.21875,0.42188 -0.28125,0.54688 -0.0937,0 -0.125,0.0469 -0.0156,0.0469 -0.0156,0.14063 l 0.0781,0.26562 q 0,0.15625 -0.17188,0.3125 -0.125,0.0625 -0.23437,0.23438 -0.10938,0.15625 -0.14063,0.21875 0.0625,0.0625 0.0625,0.15625 0,0.23437 -0.21875,0.57812 0,0.54688 -0.45312,1.625 0.0312,0.57813 0.0625,0.75 0.0312,0.15625 0.15625,0.125 h 0.15625 l 0.70312,-0.0312 q 0.23438,-0.0937 0.10938,0.48438 -0.10938,0.5625 -0.54688,1.75 -0.48437,0.73437 -1.01562,1.29687 -0.51563,0.5625 -0.6875,0.5625 z m 8.84204,-3.14062 q -0.57813,-0.0937 -1.0625,-0.64063 -0.48438,-0.54687 -0.48438,-1.25 0,-0.76562 0.96875,-3.9375 0.15625,-0.82812 0.70313,-2.14062 0.1875,-0.54688 0.21875,-0.64063 0.57812,-1.92187 1.57812,-4.70312 l 0.28125,-0.76563 q 1.125,-3.04687 1.34375,-3.71875 l 0.51563,-1.40625 q 0,-0.25 0.35937,-0.48437 0.375,-0.25 0.625,-0.25 0.51563,0 0.98438,0.67187 0.48437,0.67188 0.51562,1.40625 0,0.39063 -0.15625,0.70313 -0.73437,1.60937 -1.82812,4.96875 -0.46875,1.40625 -0.95313,2.67187 -0.48437,1.26563 -0.76562,1.8125 l -0.35938,0.70313 -0.4375,1.6875 q 0,0.0937 -0.0312,0.17187 -0.0312,0.0625 -0.0312,0.0937 0,0.0937 0.0312,0.0937 l 0.53125,-0.54687 q 0.9375,-0.92188 2.21875,-1.89063 1.04688,-0.73437 1.07813,-0.82812 0.0312,-0.0937 0.26562,-0.29688 l 0.3125,-0.25 q 1.03125,-0.73437 2.21875,-0.73437 0.34375,0 0.82813,0.35937 0.48437,0.34375 0.8125,0.96875 0.34375,0.625 0.34375,1.32813 v 0.25 0.23437 q 0,0.79688 -0.3125,2.09375 -0.29688,1.29688 -0.32813,1.48438 0.15625,0 0.46875,-0.0625 0.48438,0 0.6875,0.0937 0.21875,0.0937 0.375,0.26562 0.15625,0.15625 0.28125,0.1875 0.0625,0.0625 0.0625,0.1875 0,0.73438 -0.79687,1.28125 -0.79688,0.54688 -1.53125,0.60938 -0.89063,0 -1.4375,-0.67188 -0.54688,-0.67187 -0.54688,-1.64062 0,-0.375 0.0312,-0.75 0.0312,-0.39063 0.0625,-0.6875 0.0937,-0.64063 0.0937,-1.01563 0,-0.42187 -0.0781,-0.5625 -0.0781,-0.14062 -0.26563,-0.14062 -0.64062,0 -2.23437,1.17187 -1.57813,1.15625 -3.20313,3.01563 -1.03125,1.5 -1.95312,1.5 z m 17.8548,1.21875 q -1.82813,0 -2.95313,-1.01563 -1.10937,-1.03125 -1.10937,-2.92187 0,-0.73438 0.0937,-1.625 0.0625,-0.78125 0.625,-2.125 0.5625,-1.34375 1.53125,-2.60938 0.96875,-1.28125 2.28125,-1.89062 0.60937,-0.29688 1.03125,-0.40625 0.42187,-0.10938 0.95312,-0.10938 1.3125,0 2.125,0.48438 0.82813,0.46875 1.15625,1.07812 0.34375,0.60938 0.34375,1.0625 v 0.15625 0.0937 q 0,0.39063 -0.45312,1.42188 -0.45313,1.01562 -1.42188,1.92187 -0.96875,0.89063 -2.51562,0.98438 -0.76563,0 -1.48438,-0.14063 -0.71875,-0.15625 -0.90625,-0.15625 -0.48437,0 -0.60937,0.35938 -0.125,0.34375 -0.125,1.1875 0.0937,0.89062 0.48437,1.25 0.40625,0.34375 1.17188,0.34375 0.39062,0 0.89062,-0.21875 0.51563,-0.26563 1.21875,-0.8125 0.39063,-0.3125 0.60938,-0.4375 0.21875,-0.125 0.54687,-0.125 0.28125,0 0.67188,0.39062 0.39062,0.375 0.42187,0.71875 -0.0781,0.39063 -0.71875,1.15625 -0.625,0.76563 -1.65625,1.375 -1.03125,0.60938 -2.20312,0.60938 z m 1.10937,-7.71875 q 0.60938,0 0.84375,-0.125 0.3125,-0.15625 0.96875,-0.8125 0.65625,-0.65625 0.6875,-0.90625 0,-0.32813 -0.28125,-0.53125 -0.26562,-0.20313 -0.73437,-0.28125 h -0.35938 q -0.42187,0 -1.20312,0.70312 -0.78125,0.6875 -1.35938,1.57813 0.0312,0.15625 0.48438,0.26562 0.45312,0.10938 0.95312,0.10938 z m 16.38513,7.90625 q -0.98437,0 -1.85937,-0.4375 -0.85938,-0.45313 -1.375,-1.14063 -0.5,-0.6875 -0.5,-1.35937 0,-0.54688 0.4375,-0.54688 0.29687,0.0625 0.67187,0.35938 0.48438,0.28125 0.8125,0.4375 0.34375,0.15625 0.70313,0.15625 0.53125,0 0.9375,-0.17188 0.40625,-0.17187 0.46875,-0.53125 0,-0.125 -0.14063,-0.54687 -0.125,-0.4375 -0.3125,-0.79688 -0.70312,-1.04687 -0.95312,-2.04687 -0.23438,-1.01563 -0.29688,-2.35938 0,-0.73437 1.0625,-1.75 1.07813,-1.03125 2.5,-1.82812 1.4375,-0.8125 2.32813,-0.90625 1.76562,0 2.10937,0.29687 0.45313,0.34375 0.82813,1.14063 0.39062,0.79687 0.39062,1.34375 0,0.23437 -0.21875,0.57812 -0.20312,0.32813 -0.26562,0.42188 l -0.28125,0.4375 -0.21875,0.48437 q -0.0781,0.15625 -0.21875,0.48438 -0.14063,0.3125 -0.25,0.39062 -0.10938,0.0625 -0.34375,0.0156 -0.53125,0 -0.98438,-0.46875 -0.45312,-0.48438 -0.45312,-1.0625 0,-0.40625 0.28125,-0.82813 0.29687,-0.42187 0.29687,-0.48437 0,-0.125 -0.28125,-0.125 -0.42187,0 -1.29687,0.42187 -0.85938,0.40625 -1.53125,1.09375 -0.67188,0.6875 -0.67188,1.53125 0,0.4375 0.17188,1.03125 0.1875,0.59375 0.40625,0.98438 0.26562,0.25 0.75,1.51562 0.5,1.26563 0.5,2 0,1.125 -0.84375,1.70313 -0.84375,0.5625 -2.35938,0.5625 z m 11.04377,-0.64063 q -0.3125,0.0625 -0.57813,0.0625 -0.40625,0 -0.76562,-0.0781 -0.34375,-0.0781 -0.34375,-0.20313 0,-0.0937 -0.0469,-0.125 -0.0312,-0.0312 -0.125,-0.0312 -0.0937,-0.0937 -0.4375,-0.32812 -0.32812,-0.25 -0.54687,-0.34375 -0.39063,-0.54688 -0.39063,-1.1875 0,-0.60938 0.39063,-1.95313 0.0312,-1.25 0.90625,-2.79687 0.875,-1.54688 2.125,-2.875 1.25,-1.32813 2.25,-1.90625 0.46875,-0.48438 1.34375,-0.48438 0.67187,0 1.1875,0.28125 0.40625,0.26563 1.03125,1.09375 0.625,0.82813 0.9375,1.5 0.0781,0.23438 0.1875,0.625 0.10937,0.375 0.10937,0.92188 l -0.0625,0.89062 q -0.39062,1.3125 -0.39062,2.35938 -0.0625,0.45312 0.0625,0.73437 0.14062,0.26563 0.35937,0.26563 0.15625,0 0.34375,-0.0937 0.20313,-0.0937 0.45313,-0.0937 0.42187,0 0.70312,0.25 0.29688,0.25 0.29688,0.64062 0,0.3125 -0.17188,0.60938 -0.28125,0.64062 -1.0625,1.15625 -0.78125,0.5 -1.45312,0.5 -0.54688,0 -1.04688,-0.5625 -0.5,-0.57813 -0.78125,-1.51563 l -0.1875,-0.73437 -1.15625,1.0625 -0.73437,0.73437 q -0.51563,0.51563 -0.57813,0.60938 -0.15625,0.25 -0.5625,0.48437 -0.40625,0.21875 -1.26562,0.53125 z m 0.85937,-3.0625 q 0.57813,-0.32812 0.96875,-0.79687 0.40625,-0.48438 1.01563,-1.34375 0.14062,-0.20313 0.26562,-0.40625 0.125,-0.20313 0.3125,-0.4375 0.54688,-0.73438 0.78125,-1.17188 0.25,-0.45312 0.375,-0.96875 -0.125,-0.42187 -0.375,-0.78125 -0.23437,-0.375 -0.42187,-0.40625 -0.90625,0 -2.4375,2.10938 -0.15625,0.23437 -0.5,0.8125 -0.32813,0.5625 -0.42188,0.76562 -0.17187,0.21875 -0.4375,1.07813 -0.26562,0.84375 -0.26562,1.09375 0,0.70312 0.40625,0.70312 0.32812,0 0.73437,-0.25 z m 25.32318,0.92188 q 0.21875,0.32812 0.21875,0.60937 0,0.32813 -0.23437,0.57813 -0.21875,0.25 -0.375,0.25 -0.76563,0 -1.71875,-0.35938 -0.9375,-0.375 -0.9375,-1.07812 v -1.98438 q 0,-1.28125 -0.21875,-1.46875 -0.48438,-0.25 -1.28125,0.71875 -0.79688,0.96875 -1.95313,2.89063 -0.45312,0.70312 -0.78125,1.01562 -0.3125,0.29688 -0.5625,0.29688 -0.21875,0 -0.39062,-0.125 -0.98438,-0.28125 -0.98438,-1.125 0,-0.0937 0.0625,-0.40625 0.28125,-1 0.60938,-2.20313 0.34375,-1.21875 0.54687,-2.09375 -0.0937,-0.1875 -0.32812,-0.1875 -0.3125,0 -0.73438,0.32813 -0.125,0.0937 -0.46875,0.29687 -0.32812,0.20313 -0.67187,0.54688 -0.32813,0.32812 -0.6875,0.90625 -0.125,0.23437 -0.4375,0.67187 -1.60938,2.4375 -1.95313,2.75 -0.17187,0.45313 -0.3125,0.64063 -0.14062,0.17187 -0.32812,0.17187 -0.125,0 -0.32813,-0.10937 -0.60937,-0.28125 -0.98437,-0.8125 -0.35938,-0.53125 -0.35938,-1.26563 0,-0.46875 0.28125,-1.90625 0.29688,-1.45312 0.65625,-2.70312 0.25,-0.5 0.39063,-1.0625 0.14062,-0.5625 0.14062,-0.78125 0.10938,-0.60938 0.39063,-1.64063 0.28125,-1.04687 0.42187,-1.15625 0.1875,-0.1875 0.70313,-0.1875 0.34375,0 0.65625,0.14063 0.32812,0.125 0.45312,0.375 0.42188,0.48437 0.42188,1.3125 0,0.76562 -0.3125,1.625 -0.32813,0.73437 -0.8125,2.15625 0,0 0.0625,-0.0781 0.78125,-1.07812 1.98437,-2.42187 0.57813,-0.64063 1.29688,-0.98438 0.71875,-0.35937 1.17187,-0.35937 0.76563,0.125 1.29688,0.75 0.53125,0.625 0.65625,1.70312 0,0.39063 -0.0625,0.95313 -0.0625,0.5625 -0.0937,0.71875 0.125,-0.15625 0.65625,-0.78125 0.53125,-0.625 1.29687,-1.29688 0.70313,-0.54687 1.3125,-0.54687 0.5,0 0.89063,0.42187 0.70312,0.40625 1.09375,1.34375 0.40625,0.92188 0.40625,2.10938 0,0.40625 -0.0312,0.64062 -0.0312,0.21875 -0.0312,0.64063 0,0.4375 0.0937,0.84375 0.10937,0.40625 0.20312,0.71875 z m 4.44238,2.3125 q -1.0625,-0.42188 -1.5625,-1.29688 -0.48437,-0.89062 -0.48437,-2.29687 0,-0.3125 0.0625,-1.15625 0.0312,-0.25 0.0312,-0.59375 0,-0.54688 -0.125,-0.8125 -0.125,-0.28125 -0.45313,-0.57813 -0.0937,-0.0937 -0.45312,-0.40625 -0.34375,-0.32812 -0.3125,-0.48437 v -0.0937 q 0,-0.21875 0.10937,-0.48438 0.10938,-0.28125 0.21875,-0.34375 0.0937,-0.0312 0.375,-0.1875 0.29688,-0.17187 0.45313,-0.10937 0.125,0 0.39062,0.17187 0.28125,0.15625 0.28125,0.28125 0.1875,0.39063 0.98438,-1.25 0.45312,-0.67187 1.21875,-1.32812 0.76562,-0.65625 1.40625,-1 0.23437,-0.10938 1.01562,-0.28125 0.78125,-0.17188 1.23438,-0.17188 0.5,0 1.1875,0.42188 0.6875,0.40625 1.14062,0.89062 0.67188,0.48438 0.67188,1.82813 0,0.79687 -0.25,1.375 -0.125,0.46875 -1.26563,1.45312 -1.14062,0.96875 -1.84375,1.26563 -0.70312,0.15625 -1.5625,0.15625 -0.875,0 -1.1875,-0.15625 -0.21875,-0.125 -0.375,0.10937 -0.14062,0.23438 -0.23437,1.07813 l -0.0312,0.3125 q 0,0.35937 0.15625,0.67187 0.23437,0.39063 0.42187,0.54688 0.1875,0.15625 0.57813,0.15625 l 0.28125,-0.0312 q 0.35937,-0.0312 0.75,-0.34375 0.40625,-0.32813 0.95312,-0.875 0.57813,-0.54688 0.67188,-0.64063 0.15625,-0.3125 0.85937,-0.3125 0.28125,0 0.73438,0.0937 0.32812,0.25 0.32812,0.64063 0,0.54687 -0.51562,1.46875 -0.15625,0.32812 -0.98438,1.09375 -0.8125,0.76562 -1.29687,1.04687 -0.85938,0.48438 -1.90625,0.48438 -0.8125,0 -1.67188,-0.3125 z m 2.375,-7.20313 q 0.76563,-0.20312 1.1875,-0.46875 0.4375,-0.26562 1.01563,-1.07812 0.25,-0.76563 0.0937,-0.92188 -0.1875,-0.1875 -0.45313,-0.1875 -0.3125,0 -0.76562,0.34375 -0.3125,0.0937 -0.875,0.59375 -0.5625,0.5 -0.95313,0.96875 -0.375,0.45313 -0.21875,0.54688 -0.0625,0.10937 0.3125,0.15625 0.39063,0.0469 0.65625,0.0469 z"
id="path4094"
inkscape:connector-curvature="0"
style="fill:#000000;fill-rule:nonzero" />
<path
d="M 427.809,138.3307 H 553.08723 V 253.13125 H 427.809 Z"
id="path4100"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="M 379.751,138.43701 H 505.04235 V 253.24016 H 379.751 Z"
id="path4109"
inkscape:connector-curvature="0"
style="fill:#00fdc8;fill-rule:evenodd" />
<path
d="M 379.751,138.43701 H 505.04235 V 253.24016 H 379.751 Z"
id="path4111"
inkscape:connector-curvature="0"
style="fill-rule:evenodd;stroke:#666666;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
<path
d="m 299.38583,404.37924 h 70.36065 v -6.68504 l 8.40525,13.37008 -8.40525,13.37007 v -6.68502 h -70.36065 z"
id="path4071-3"
inkscape:connector-curvature="0"
style="fill:#7c7ce0;fill-rule:evenodd;stroke-width:0.79288208" />
<path
d="m 506.71874,190.09462 h 70.36065 v -6.68504 l 8.40525,13.37008 -8.40525,13.37007 v -6.68502 h -70.36065 z"
id="path4071-6"
inkscape:connector-curvature="0"
style="fill:#7c7ce0;fill-rule:evenodd;stroke-width:0.79288208" />
<path
d="m 506.71875,404.41666 h 70.36065 v -6.68504 l 8.40525,13.37008 -8.40525,13.37007 v -6.68502 h -70.36065 z"
id="path4071-6-7"
inkscape:connector-curvature="0"
style="fill:#7c7ce0;fill-rule:evenodd;stroke-width:0.79288208" />
<flowRoot
xml:space="preserve"
id="flowRoot4272"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:26.66666603px;line-height:1.25;font-family:'Comic Sans MS';-inkscape-font-specification:'Comic Sans MS';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none"><flowRegion
id="flowRegion4274"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:26.66666603px;font-family:'Comic Sans MS';-inkscape-font-specification:'Comic Sans MS';text-align:center;text-anchor:middle"><rect
id="rect4276"
width="125.97456"
height="90.782555"
x="379.06781"
y="162.45763"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:26.66666603px;font-family:'Comic Sans MS';-inkscape-font-specification:'Comic Sans MS';text-align:center;text-anchor:middle" /></flowRegion><flowPara
id="flowPara4278">Some process</flowPara></flowRoot> <g
transform="matrix(0.45555722,0,0,0.45555774,379.751,138.43701)"
id="g4082-1"
inkscape:transform-center-x="-34.322034"
inkscape:transform-center-y="102.9661">
<clipPath
id="clipPath4292">
<path
d="M 0,0 H 275 V 252 H 0 Z"
id="path4290"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
<image
clip-path="url(#p.3-4)"
width="275"
height="252"
x="0"
y="0"
preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/magic.gif"
id="image4080-3"
style="fill:#000000" />
</g>
<rect
style="fill:#ffffff"
id="rect148"
width="28.983051"
height="45.762711"
x="212.03391"
y="388.22034" />
</g>
</svg>
</center></p>
<p>Let's now translate that into a formal definition. <a name="definition"></a></p>
<p>A process <span class="math">\(A\)</span> is <span class="math">\(\varepsilon\)</span>-differentially private if for all databases <span class="math">\(D_1\)</span>
and <span class="math">\(D_2\)</span> which differ in only one individual:</p>
<div class="math">$$
\mathbb{P}\left[A(D_1)=O\right] \le e^\varepsilon\cdot\mathbb{P}\left[A(D_2)=O\right]
$$</div>
<p style="text-indent: 0em">… and this must be true for all
possible outputs <span class="math">\(O\)</span>. Let's unpack this.</p>
<p><span class="math">\(\mathbb{P}\left[A(D_1)=O\right]\)</span> is the probability that when you run the process <span class="math">\(A\)</span> on
the database <span class="math">\(D_1\)</span>, the output is <span class="math">\(O\)</span>. This process is probabilistic: if you run
it several times, it might give you different answers. A typical process might
be: "count the people with blue eyes, add some random number to this count, and
return this sum". Since the random number changes every time you run the
process, the results will vary.</p>
<p><span class="math">\(e^\varepsilon\)</span> is the <a href="https://en.wikipedia.org/wiki/Exponential_function">exponential function</a> applied to the parameter
<span class="math">\(\varepsilon>0\)</span>. If <span class="math">\(\varepsilon\)</span> is very close to 0, then <span class="math">\(e^\varepsilon\)</span> is very close
to 1, so the probabilities are very similar. The bigger <span class="math">\(\varepsilon\)</span> is, the more
the probabilities can differ.</p>
<p>Of course, the definition is symmetrical: you can replace <span class="math">\(D_1\)</span> by <span class="math">\(D_2\)</span> and
vice-versa, and the two databases will still differ in only one individual. So
we could replace it by:</p>
<div class="math">$$
e^{-\varepsilon}\cdot\mathbb{P}\left[A(D_2)=O\right]
\le \mathbb{P}\left[A(D_1)=O\right] \le
e^\varepsilon\cdot\mathbb{P}\left[A(D_2)=O\right]
$$</div>
<p>Thus, this formula means that the output of the process is similar if you change
or remove the data of one person. The degree of similarity depends on
<span class="math">\(\varepsilon\)</span>: the smaller it is, the more similar the outputs are.</p>
<p>What does this similarity have to do with privacy? First, I'll explain this with
an intuitive example. Then, I'll formalize this idea with a more generic
interpretation.</p>
<h1 id="a-simple-example-randomized-response">A simple example: randomized response <a name="rr"></a></h1>
<p>Suppose you want to do a survey to know how many people are illegal drug users.
If you naively go out and ask people whether they're using illegal drugs, many
will lie to you. So you devise the following mechanism. The participants no
longer directly answer the question "have you consumed illegal drugs in the past
week?". Instead, each of them will flip a coin, without showing it to you.</p>
<ul>
<li>On heads, the participant tells the truth (<em>Yes</em> or <em>No</em>).</li>
<li>On tails, they flip a second coin. If the second coin lands on heads, they
answer <em>Yes</em>. Otherwise, they answer <em>No</em>.</li>
</ul>
<p>How is this better for survey respondents? They can now answer <em>Yes</em> without
revealing that they're doing something illegal. When someone answers <em>Yes</em>, you
can't know their true answer for sure. They could be actually doing drugs, but
they might also have answered at random. </p>
<p>Let's compute the probabilities of each answer for a drug user.</p>
<ul>
<li>With probability 50%, they will say the truth and answer <em>Yes</em>.</li>
<li>With probability 50%, they will answer at random.<ul>
<li>They then have another 50% chance to answer <em>Yes</em>, so 25% chance in total.</li>
<li>Similarly, in total, they have a 25% chance to answer <em>No</em>.</li>
</ul>
</li>
</ul>
<p>All in all, we get a 75% chance to answer <em>Yes</em> and a 25% chance to answer <em>No</em>.
For someone who is not doing drugs, the probabilities are reversed: 25% chance
to answer <em>Yes</em> and 75% to answer <em>No</em>. Using the notations from earlier:</p>
<ul>
<li><span class="math">\(\mathbb{P}\left[A(Yes)=Yes\right] = 0.75\)</span>, <span class="math">\(\mathbb{P}\left[A(Yes)=No\right] = 0.25\)</span></li>
<li><span class="math">\(\mathbb{P}\left[A(No)=Yes\right] = 0.25\)</span>, <span class="math">\(\mathbb{P}\left[A(No)=No\right] = 0.75\)</span></li>
</ul>
<p>Now, <span class="math">\(0.75\)</span> is three times bigger than <span class="math">\(0.25\)</span>. So if we choose <span class="math">\(\varepsilon\)</span>
such as <span class="math">\(e^\varepsilon=3\)</span> (that's <span class="math">\(\varepsilon\simeq1.1\)</span>), this process is
<span class="math">\(\varepsilon\)</span>-differentially private. So this plausible deniability translates
nicely in the language of differential privacy.</p>
<p>Of course, with a differentially private process like this one, you're getting
some noise into your data. But if you have enough answers, with high
probability, the noise will cancel itself out. Suppose you have 1000 answers in
total: 400 of them are <em>Yes</em> and 600 are <em>No</em>. About 50% of all 1000 answers are
random, so you can remove 250 answers from each count. In total, you get 150
<em>Yes</em> answers out of 500 non-random answers, so about 30% of <em>Yes</em> overall.</p>
<p>What if you want more privacy? Instead of having the participants say the truth
with probability 50%, you can have them tell the truth 25% of the time. What if
you want less noise instead, at the cost of less protection? Have them tell the
truth 75% of the time. Finding out <span class="math">\(\varepsilon\)</span> and quantifying the noise for
each option is left as an exercise for the reader =)</p>
<h1 id="a-generalization-quantifying-the-attackers-knowledge">A generalization: quantifying the attacker's knowledge</h1>
<p><a name="quantifying"></a></p>
<p>Let's forget about the previous example and consider a more generic scenario. In
line with the <a href="differential-privacy-awesomeness.html">previous article</a>, we will describe this scenario from the
<em>attacker</em>'s perspective. We have a mechanism <span class="math">\(A\)</span> which is
<span class="math">\(\varepsilon\)</span>-differentially private. We run it on some database <span class="math">\(D\)</span>, and
release the output <span class="math">\(A(D)\)</span> to an attacker. Then, the attacker tries to figure out
whether someone (their <em>target</em>) is in <span class="math">\(D\)</span>.</p>
<p>Under differential privacy, the attacker can't gain a lot of information about
their target. And this is true even if this attacker has a lot of knowledge
about the dataset. Let's take the stronger attacker we can think of: they know
<em>all the database, except their target</em>. This attacker has to determine which
database is the real one, between two options: one with their target in it
(let's call it <span class="math">\(D_{in}\)</span>), the other without (<span class="math">\(D_{out}\)</span>)<sup id="fnref:dbs"><a class="footnote-ref" href="#fn:dbs">1</a></sup>.</p>
<p>So, in the attacker's model of the world, the actual database <span class="math">\(D\)</span> can be either
<span class="math">\(D_{in}\)</span> or <span class="math">\(D_{out}\)</span>. They might have an <em>initial suspicion</em> that their target
is in the database. This suspicion is represented by a <em>probability</em>,
<span class="math">\(\mathbb{P}\left[D=D_{in}\right]\)</span>. This probability can be anything between <span class="math">\(0\)</span> and <span class="math">\(1\)</span>.
Say, <span class="math">\(0.9\)</span> if the attacker's suspicion is strong, <span class="math">\(0.01\)</span> if they think it's very
unlikely, <span class="math">\(0.5\)</span> if they have no idea… Similarly, their suspicion that their
target is <em>not</em> in the dataset is also a probability, <span class="math">\(\mathbb{P}\left[D=D_{out}\right]\)</span>.
Since there are only two options,
<span class="math">\(\mathbb{P}\left[D=D_{out}\right]=1-\mathbb{P}\left[D=D_{in}\right]\)</span>.</p>
<p>Now, suppose the attacker sees that the mechanism returns output <span class="math">\(O\)</span>. How much
information did the attacker gain? This is captured by looking at how much their
suspicion changed after seeing this output. In mathematical terms, we have to
compare <span class="math">\(\mathbb{P}\left[D=D_{in}\right]\)</span> with the updated suspicion
<span class="math">\(\mathbb{P}\left[D=D_{in}\mid A(D)=O\right]\)</span>. This updated suspicion is the attacker's
model of the world after seeing <span class="math">\(O\)</span>.</p>
<p>With differential privacy, the updated probability is never too far from the
initial suspicion. And we can quantify this phenomenon exactly. For example,
with <span class="math">\(\varepsilon=1.1\)</span>, here is what the upper and lower bounds look like.</p>
<p><center>
<img alt="Graph showing the bounds on the posterior as a function of the prior" src="https://desfontain.es/privacy/images/dp-bounds-graph.svg">
</center></p>
<p>The black line is what happens if the attacker didn't get their suspicion
updated at all. The blue lines are the lower and upper bounds on the updated
suspicion: it can be anywhere between the two. We can visualize the example
mentioned in the <a href="differential-privacy-awesomeness.html">previous article</a>: for an initial suspicion of 50%, the
updated suspicion is approximately between 25% and 75%.</p>
<p>How do we prove that these bounds hold? We'll need a result from probability
theory, and some basic arithmetic manipulation. I reproduced the proof as simply
as I could, but you still don't have to read it. If you want to, click here:
<button id="toggleProof"></button></p>
<div id="proof" style="display: none; border-left: double; padding-left: 10px">
<p>The proof is based on a theorem called Bayes' rule. Explaining the full
intuition behind this theorem is a bit out of scope for this post. If you want
to understand what it says and why it works, I recommend you read <a href="https://arbital.com/p/bayes_rule/?l=1zq">this
guide</a>. If you don't, just trust me: this theorem allows us to rephrase
the updated suspicion in other terms.</p>
<p>
<div class="math">$$
\mathbb{P}\left[D=D_{in}\mid A(D)=O\right]=\frac{\mathbb{P}\left[D=D_{in}\right]\cdot\mathbb{P}\left[A(D)=O\mid D=D_{in}\right]}{\mathbb{P}\left[A(D)=O\right]}
$$</div>
</p>
<p>Let's interpret each of these terms. We recognize <span class="math">\(\mathbb{P}\left[D=D_{in}\right]\)</span>, that's
the initial suspicion of the attacker. <span class="math">\(\mathbb{P}\left[A(D)=O\mid D=D_{in}\right]\)</span> is the
probability of getting output <span class="math">\(O\)</span> from database <span class="math">\(D_{in}\)</span>, which we can simplify
into <span class="math">\(\mathbb{P}\left[A\left(D_{in}\right)=O\right]\)</span>. Finally, <span class="math">\(\mathbb{P}\left[A(D)=O\right]\)</span> is the
probability that we get the output <span class="math">\(O\)</span>, in the attacker's model of the world.</p>
<p>This last term is icky. We don't know its value. So let's make it disappear by
considering the <em>ratio</em> between the two updated probabilities
<span class="math">\(\mathbb{P}\left[D=D_{in}\mid A(D)=O\right]\)</span> and <span class="math">\(\mathbb{P}\left[D=D_{out}\mid A(D)=O\right]\)</span>. The
icky term will nicely go away:</p>
<p>
<div class="math">$$
\frac{\mathbb{P}\left[D=D_{in}\mid A(D)=O\right]}{\mathbb{P}\left[D=D_{out}\mid A(D)=O\right]} =
\frac{\mathbb{P}\left[D=D_{in}\right]}{\mathbb{P}\left[D=D_{out}\right]}
\cdot\frac{\mathbb{P}\left[A\left(D_{in}\right)=O\right]}{\mathbb{P}\left[A\left(D_{out}\right)=O\right]}
$$</div>
</p>
<p>Note that this isn't a meaningless math trick. This ratio of probabilities
actually has a simple interpretation: it's what gamblers call <em>betting odds</em>.
For example, on betting websites, the odds for the France v. Croatia game in the
2018 World Cup were 2:1. This means that according to bookies, the probability
for France to win was twice as much as for Croatia. This corresponds to
probabilities of about 67% and 33%, respectively.</p>
<p>Anyway, look! The two terms <span class="math">\(\mathbb{P}\left[A\left(D_{in}\right)=O\right]\)</span> and
<span class="math">\(\mathbb{P}\left[A\left(D_{out}\right)=O\right]\)</span> are the ones from the differential privacy
definition. So we know that their ratio is bounded:</p>
<p>
<div class="math">$$
e^{-\varepsilon} \le
\frac{\mathbb{P}\left[A\left(D_{in}\right)=O\right]}{\mathbb{P}\left[A\left(D_{out}\right)=O\right]}
\le e^\varepsilon
$$</div>
</p>
<p>If we plug this into the previous formula, we get a nice relation:</p>
<p>
<div class="math">$$
e^{-\varepsilon}\cdot\frac{\mathbb{P}\left[D=D_{in}\right]}{\mathbb{P}\left[D=D_{out}\right]} \le
\frac{\mathbb{P}\left[D=D_{in}\mid A(D)=O\right]}{\mathbb{P}\left[D=D_{out}\mid A(D)=O\right]} \le
e^\varepsilon\cdot\frac{\mathbb{P}\left[D=D_{in}\right]}{\mathbb{P}\left[D=D_{out}\right]}
$$</div>
</p>
<p>This relation is a reformulation of differential privacy. The original
definition said that the probability distributions of outputs are similar. This
relation says that the <em>odds</em> don't change too much after looking at the output.
And the two formulations are equivalent: you could write the same proof in the
other direction.</p>
<p>But back to our proof. All we need to do now is replace <span class="math">\(\mathbb{P}\left[D=D_{out}\right]\)</span>
with <span class="math">\(1-\mathbb{P}\left[D=D_{in}\right]\)</span>, do the same for
<span class="math">\(\mathbb{P}\left[D=D_{out}\mid A\left(D\right)=O\right]\)</span>, and solve for
<span class="math">\(\mathbb{P}\left[D=D_{in}\mid A\left(D\right)=O\right]\)</span>. You end up with the
following bounds:</p>
<p>
<div class="math">$$
\frac{\mathbb{P}\left[D=D_{in}\right]}{e^{\varepsilon}+\left(1-e^{\varepsilon}\right)\cdot\mathbb{P}\left[D=D_{in}\right]}
\leq \mathbb{P}\left[D=D_{in}\mid A\left(D\right)=O\right] \leq
\frac{e^{\varepsilon}\cdot\mathbb{P}\left[D=D_{in}\right]}{1+\left(e^{\varepsilon}-1\right)\cdot\mathbb{P}\left[D=D_{in}\right]}
$$</div>
</p>
<p style="text-indent: 0em">which you can plot using your favorite software.</p>
</div>
<p>What does this look like for various values of <span class="math">\(\varepsilon\)</span>? We can draw a
generalization of this graph with pretty colors:</p>
<p><center>
<img alt="Graph showing the bounds on the posterior as a function of the prior for many values of ε" src="https://desfontain.es/privacy/images/dp-contour-graph.png">
</center></p>
<p>For larger values of <span class="math">\(\varepsilon\)</span>, this gets scary quite fast. Let's say you're
using <span class="math">\(\varepsilon=5\)</span>. Then, an attacker can go from a small suspicion (say,
10%) to a very high degree of certainty (94%).</p>
<h1 id="what-about-composition">What about composition?</h1>
<p>In the previous section, I formalized two claims I made in my <a href="differential-privacy-awesomeness.html">last article</a>.
First, I explained what it means to <em>quantify</em> information gain. Furthermore, I
picked an attacker with <em>full background knowledge</em>. If the attacker knows less
information in the first place, the bounds we showed still hold.</p>
<p>What about the third claim? I said that differential privacy was <em>composable</em>.
Suppose that two algorithms <span class="math">\(A\)</span> and <span class="math">\(B\)</span> are <span class="math">\(\varepsilon\)</span>-differentially
private. We want to prove that publishing the result of both is
<span class="math">\(2\varepsilon\)</span>-differentially private. Let's call <span class="math">\(C\)</span> the algorithm which
combines <span class="math">\(A\)</span> and <span class="math">\(B\)</span>: <span class="math">\(C(D)=\left(A(D),B(D)\right)\)</span>. The output of this
algorithm will be a pair of outputs: <span class="math">\(O=\left(O_A,O_B\right)\)</span>.</p>
<p>The insight is that the two algorithms are <em>independent</em>. They each have their
own randomness, so the result of one does not impact the result of the other.
This allows us to simply write:</p>
<div class="math">$$
\begin{align*}
\mathbb{P}\left[C\left(D_{1}\right)=O\right] & =\mathbb{P}\left[A\left(D_{1}\right)=O_{A}\right]\cdot\mathbb{P}\left[B\left(D_{1}\right)=O_{B}\right]\\
& \leq e^{2\varepsilon}\cdot\mathbb{P}\left[A\left(D_{2}\right)=O_{A}\right]\cdot\mathbb{P}\left[B\left(D_{2}\right)=O_{B}\right]\\
& \leq e^{2\varepsilon}\cdot\mathbb{P}\left[C\left(D_{2}\right)=O\right]
\end{align*}
$$</div>
<p style="text-indent: 0em">so <span class="math">\(C\)</span> is <span class="math">\(2\varepsilon\)</span>-differentially
private.</p>
<h1 id="future-steps">Future steps</h1>
<p>I hope that I convinced you that differential privacy can be an excellent way to
protect your data (if your <span class="math">\(\varepsilon\)</span> is low). Now, if everything is going
according to my master plan, you should be like… "This is awesome! I want to use
it everywhere! How do I do that?"</p>
<p>I have good news for you: this blog post has sequels. Head over to the <a href="friendly-intro-to-differential-privacy.html">table of
contents</a> of this series to decide which one you want to read next!</p>
<hr>
<p><small>Thanks to Chao Li for introducing me to the Bayesian interpretation of
differential privacy, and to <a href="http://a3nm.net/">a3nm</a>, <a href="https://virgile.anbuco.fr/">Armavica</a>, <a href="http://www.normalesup.org/~bouya/">immae</a> and <a href="https://pablo.rauzy.name/">p4bl0</a> for their
helpful comments on drafts of this article (as well as previous ones).</small></p>
<script type="text/javascript">
var button = document.getElementById('toggleProof');
var defaultButton = 'Show me the proof';
button.innerHTML = defaultButton
button.addEventListener('click', function (event) {
button.innerHTML = button.innerHTML == defaultButton ? 'Hide the proof' : defaultButton;
proof = document.getElementById('proof');
proof.style.display = proof.style.display == 'none' ? 'block' : 'none';
});
</script>
<div class="footnote">
<hr>
<ol>
<li id="fn:dbs">
<p>This can mean that <span class="math">\(D_{out}\)</span> is the same as <span class="math">\(D_{in}\)</span> with one fewer
user. This can also mean that <span class="math">\(D_{out}\)</span> is the same as <span class="math">\(D_{in}\)</span>, except one
user has been changed to some arbitrary other user. This distinction doesn't
change anything to the reasoning, so we can simply forget about it. <a class="footnote-backref" href="#fnref:dbs" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Why differential privacy is awesome2018-07-30T00:00:00+02:002021-09-27T00:00:00+02:00Damien Desfontainestag:desfontain.es,2018-07-30:/privacy/differential-privacy-awesomeness.html<p>A high-level, non-technical explanation of differential privacy and its advantages.</p><p><small>
<span class='notlettrine'>T</span>his is the first blog post in a <a href="friendly-intro-to-differential-privacy.html">series about
differential privacy</a>. Check out the <a href="friendly-intro-to-differential-privacy.html">table of contents</a> to see
the next articles!</p>
<p></small></p>
<hr>
<p><span class='lettrine'>How</span> to publish data about people while protecting
their privacy? This question is far from new. Statistical agencies have grappled
with it for decades. Computer scientists have proposed a <a href="k-anonymity.html">whole</a> <a href="k-map.html">bunch</a> of
<a href="l-diversity.html">creative</a> <a href="delta-presence.html">notions</a> to capture this idea. None of them was very satisfactory,
though: all these notions were shown to be broken in some circumstances. They
were also hard to apply without destroying the utility of the data.</p>
<p>This all changed in 2006, when four researchers introduced <em>differential
privacy</em>. This new notion took a novel approach to defining privacy leakage, one
that would prove much more rigorous and fruitful. So, what makes differential
privacy special? How did it get so successful in adademic circles? Why did
governments and tech companies start adopting it for their data publications?</p>
<p>This first article introducing differential privacy will attempt to answer that
question. First, we'll describe the high-level intuition behind this successful
notion. Then, we'll explain <em>why</em> it's so successful: why is it so much more
awesome than all the definitions that came before?</p>
<h1 id="the-core-idea-behind-differential-privacy">The core idea behind differential privacy</h1>
<p>Suppose you have a process that takes some database as input, and returns some
output.</p>
<p><center>
<svg
role="img"
aria-label="Diagram showing an arrow labeled "some process", going from a fake database to a cloud labeled "output"."
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
version="1.1"
viewBox="0 0 699.45947 142.76904"
stroke-miterlimit="10"
id="svg4123"
sodipodi:docname="no-magic.svg"
inkscape:version="0.92.2pre0 (973e216, 2017-07-25)"
width="699.45947"
height="142.76904"
style="fill:none;stroke:none;stroke-linecap:square;stroke-miterlimit:10">
<title>Diagram showing an arrow labeled "some process", going from a fake database to a cloud labeled "output".</title>
<desc>Diagram showing an arrow labeled "some process", going from a fake database to a cloud labeled "output".</desc>
<metadata
id="metadata4129">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
<dc:title></dc:title>
</cc:Work>
</rdf:RDF>
</metadata>
<defs
id="defs4127">
<clipPath
id="p.3-4">
<path
d="M 0,0 H 275 V 252 H 0 Z"
id="path4077-0"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
</defs>
<sodipodi:namedview
pagecolor="#ffffff"
bordercolor="#666666"
borderopacity="1"
objecttolerance="10"
gridtolerance="10"
guidetolerance="10"
inkscape:pageopacity="0"
inkscape:pageshadow="2"
inkscape:window-width="1266"
inkscape:window-height="1078"
id="namedview4125"
showgrid="false"
inkscape:zoom="1.3111111"
inkscape:cx="364.35056"
inkscape:cy="-64.033451"
inkscape:window-x="0"
inkscape:window-y="0"
inkscape:window-maximized="0"
inkscape:current-layer="g4121"
fit-margin-top="0"
fit-margin-left="0"
fit-margin-right="0"
fit-margin-bottom="0"
viewbox-x="82"
viewbox-width="650" />
<clipPath
id="p.0">
<path
d="M 0,0 H 960 V 720 H 0 Z"
id="path4036"
inkscape:connector-curvature="0"
style="clip-rule:nonzero" />
</clipPath>
<g
clip-path="url(#p.0)"
id="g4121"
transform="translate(-76.08921,-124.34908)">
<path
d="M 0,0 H 960 V 720 H 0 Z"
id="path4039"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 76.08921,124.34908 h 223.2992 V 267.11813 H 76.08921 Z"
id="path4041"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<g
transform="matrix(0.22307612,0,0,0.22307664,76.089214,124.34908)"
id="g4048">
<clipPath
id="p.1">
<path
d="M 0,0 H 1001 V 640 H 0 Z"
id="path4043"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
<image
clip-path="url(#p.1)"
width="1001"
height="640"
x="0"
y="0"
preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/quiestce.jpeg"
id="image4046"
style="fill:#000000" />
</g>
<path
d="M 76.086586,339.67847 H 299.38579 V 482.44751 H 76.086586 Z"
id="path4050"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 604.81974,178.70273 v 0 c -1.51233,-10.41394 3.45307,-20.72311 12.78925,-26.55296 9.33618,-5.82984 21.40576,-6.15791 31.08722,-0.84502 v 0 c 3.42944,-6.0551 9.70636,-10.23573 16.93207,-11.27733 7.2257,-1.04159 14.55151,1.17818 19.76159,5.98789 v 0 c 2.92145,-5.48995 8.65778,-9.17848 15.1734,-9.75675 6.51569,-0.57824 12.88855,2.03563 16.85718,6.9141 v 0 c 5.27802,-5.81934 13.67554,-8.2695 21.55884,-6.29033 7.8833,1.97917 13.83649,8.03222 15.28363,15.53996 v 0 c 6.46649,1.65271 11.8529,5.85413 14.76764,11.51872 2.91473,5.66462 3.07178,12.23673 0.43066,18.01831 v 0 c 6.36756,7.76532 7.85712,18.11251 3.91272,27.18018 -3.9444,9.06767 -12.7301,15.49351 -23.07849,16.87953 -0.0729,8.51033 -5.05414,16.31931 -13.02356,20.41701 -7.96942,4.09772 -17.68268,3.84427 -25.39587,-0.66263 -3.2854,10.1926 -12.53272,17.6922 -23.74671,19.25873 -11.21405,1.56653 -22.38446,-3.08087 -28.68524,-11.93434 -7.72339,4.3639 -16.99084,5.621 -25.71185,3.48773 -8.72095,-2.13329 -16.16077,-7.47725 -20.64124,-14.8264 v 0 c -7.89233,0.86539 -15.52319,-2.96604 -19.10534,-9.59276 -3.58216,-6.62671 -2.35303,-14.63803 3.07733,-20.05798 v 0 c -7.04022,-3.8826 -10.63251,-11.58696 -8.90375,-19.09555 1.72882,-7.50861 8.38696,-13.11998 16.50256,-13.90799 z"
id="path4059"
inkscape:connector-curvature="0"
style="fill:#f6cd4c;fill-rule:evenodd" />
<path
d="m 597.06497,212.10764 v 0 c 3.32227,1.83222 7.16028,2.66335 10.99872,2.38181 m 5.02759,27.2696 c 1.65063,-0.18098 3.26861,-0.56427 4.81219,-1.13993 m 41.53858,12.47812 c -1.16095,-1.63134 -2.133,-3.37456 -2.89954,-5.19996 m 55.3335,-2.12474 v 0 c 0.59893,-1.85826 0.98706,-3.77081 1.15777,-5.70575 m 37.26019,-14.04791 c 0.0776,-9.06056 -5.41461,-17.3565 -14.11767,-21.32448 m 33.2832,-22.73425 c -1.40942,3.08528 -3.56109,5.8222 -6.28625,7.99617 m -8.91071,-37.53403 v 0 c 0.24011,1.24591 0.35126,2.51055 0.33185,3.77639 m -37.17371,-13.02565 v 0 c -1.31659,1.45164 -2.4013,3.07382 -3.22033,4.81604 m -28.81055,-1.97402 v 0 c -0.70166,1.31851 -1.22559,2.71372 -1.55975,4.15347 m -35.13446,1.13608 v 0 c 2.04858,1.12421 3.94384,2.47731 5.64404,4.02959 m -49.5199,23.36855 v 0 c 0.20844,1.4353 0.53784,2.85292 0.98499,4.23957"
id="path4061"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 636.30257,211.8428 h -0.1875 q -0.78125,0 -1.25,-0.48437 -0.46875,-0.48438 -0.46875,-1.17188 0,-0.46875 0.5,-2.92187 l 1.375,-7.10938 q 0.46875,-2.42187 2.0625,-13.4375 l 0.39063,-2.70312 q 0.14062,-1.01563 1.10937,-1.84375 0.96875,-0.84375 1.78125,-0.84375 0.4375,0 2.9375,1.3125 2.70313,1.45312 3.20313,1.59375 5.09375,1.875 8.375,5.6875 3.29687,3.8125 3.29687,8.23437 0,4.04688 -2.40625,7.92188 -2.39062,3.875 -5.78125,5.71875 -3.39062,1.84375 -8.67187,1.84375 -1.53125,0 -3.46875,-0.53125 -1.9375,-0.51563 -2.79688,-1.26563 z m 6,-25.40625 -1.64062,11.20313 -2,10.67187 q 0.0625,0.0469 0.14062,0.0781 1.84375,1.32812 4.375,1.32812 5.01563,0 7.375,-1.79687 2.35938,-1.79688 3.60938,-4.21875 1.25,-2.42188 1.25,-4.92188 0,-2.51562 -1.40625,-4.64062 -1.39063,-2.125 -3.76563,-3.57813 -2.35937,-1.46875 -7.9375,-4.125 z m 32.16406,24.28125 q -4.21875,2.03125 -6.01562,2.03125 -7.29688,0 -7.29688,-6.95312 0,-6.46875 4.07813,-10.46875 4.09375,-4 8.98437,-4 1.89063,0 3.90625,0.96875 2.03125,0.96875 2.03125,2.25 0,0.79687 -0.53125,1.25 -0.39062,0.76562 -1.10937,4.40625 -0.70313,3.64062 -0.70313,5.25 0,1.96875 1.01563,5.42187 l 0.0781,0.3125 q -0.4375,1.82813 -2.17187,1.82813 -0.29688,0 -1.09375,-0.79688 -0.79688,-0.79687 -1.17188,-1.5 z m 1.42188,-15.1875 q -1.21875,-0.73437 -1.95313,-0.73437 -3.25,0 -6.07812,2.95312 -2.82813,2.9375 -2.82813,7.3125 0,4.29688 3.71875,4.29688 2.90625,0 5.46875,-2.15625 0.3125,-8.1875 1.67188,-11.67188 z m 16.45312,-0.34375 q -0.23437,0.9375 -0.60937,2.1875 -1.64063,5.42188 -1.64063,8.26563 0,3.35937 1.5625,3.35937 1.54688,0 2.28125,-0.95312 1.01563,-1.32813 1.875,-1.32813 0.65625,0 1.25,0.48438 0.59375,0.48437 0.59375,1.26562 0,1.71875 -2.15625,2.95313 -2.15625,1.21875 -4.29687,1.21875 -5.03125,0 -5.03125,-6.28125 0,-3.4375 1.54687,-8.39063 0.40625,-1.3125 0.78125,-2.625 -4.25,-0.20312 -4.85937,-0.70312 -0.60938,-0.5 -0.60938,-1.5625 0,-0.85938 0.57813,-1.42188 0.59375,-0.57812 1.60937,-0.57812 l 4.03125,0.32812 q 0.125,-0.51562 0.1875,-1.17187 0.0312,-0.67188 0.125,-1.48438 l 0.34375,-2.42187 q 0.10938,-0.79688 0.71875,-1.34375 0.625,-0.54688 1.46875,-0.54688 1.64063,0 1.64063,2.625 0,0.76563 -0.23438,1.98438 l -0.39062,2.35937 q 2.60937,-0.32812 2.78125,-0.32812 2.03125,0 2.79687,0.40625 0.76563,0.39062 0.76563,1.46875 0,0.95312 -0.65625,1.54687 -0.65625,0.57813 -1.57813,0.57813 l -1.92187,-0.0781 q -1.45313,0 -2.95313,0.1875 z m 21.44141,15.53125 q -4.21875,2.03125 -6.01563,2.03125 -7.29687,0 -7.29687,-6.95312 0,-6.46875 4.07812,-10.46875 4.09375,-4 8.98438,-4 1.89062,0 3.90625,0.96875 2.03125,0.96875 2.03125,2.25 0,0.79687 -0.53125,1.25 -0.39063,0.76562 -1.10938,4.40625 -0.70312,3.64062 -0.70312,5.25 0,1.96875 1.01562,5.42187 l 0.0781,0.3125 q -0.4375,1.82813 -2.17188,1.82813 -0.29687,0 -1.09375,-0.79688 -0.79687,-0.79687 -1.17187,-1.5 z m 1.42187,-15.1875 q -1.21875,-0.73437 -1.95312,-0.73437 -3.25,0 -6.07813,2.95312 -2.82812,2.9375 -2.82812,7.3125 0,4.29688 3.71875,4.29688 2.90625,0 5.46875,-2.15625 0.3125,-8.1875 1.67187,-11.67188 z"
id="path4063"
inkscape:connector-curvature="0"
style="fill:#000000;fill-rule:nonzero" />
<path
d="m 597.06237,427.43705 v 0 c 3.32227,1.83218 7.16028,2.66333 10.99872,2.38181 m 5.02759,27.26959 c 1.65063,-0.181 3.26861,-0.56427 4.81219,-1.13992 m 41.53858,12.47811 c -1.16095,-1.63134 -2.133,-3.37457 -2.89954,-5.19998 m 55.3335,-2.12472 v 0 c 0.59893,-1.85825 0.98706,-3.77082 1.15777,-5.70575 m 37.26019,-14.04792 c 0.0776,-9.06054 -5.41461,-17.3565 -14.11767,-21.32449 m 33.2832,-22.73425 c -1.40942,3.0853 -3.56109,5.82223 -6.28625,7.99618 m -8.91071,-37.53402 v 0 c 0.24011,1.24591 0.35126,2.51056 0.33185,3.77639 m -37.17371,-13.02566 v 0 c -1.31659,1.45163 -2.4013,3.07382 -3.22033,4.81604 m -28.81055,-1.974 v 0 c -0.70166,1.31848 -1.22559,2.71372 -1.55975,4.15347 m -35.13446,1.13608 v 0 c 2.04858,1.12421 3.94384,2.4773 5.64404,4.02957 m -49.5199,23.36856 v 0 c 0.20844,1.43531 0.53784,2.85291 0.98499,4.23957"
id="path4067"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 299.38846,189.04987 h 70.36065 v -6.68504 l 8.40525,13.37008 -8.40525,13.37007 v -6.68502 h -70.36065 z"
id="path4071"
inkscape:connector-curvature="0"
style="fill:#7c7ce0;fill-rule:evenodd;stroke-width:0.79288208" />
<path
d="M 427.88773,353.66272 H 553.16596 V 468.46329 H 427.88773 Z"
id="path4075"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="M 427.809,138.3307 H 553.08723 V 253.13125 H 427.809 Z"
id="path4100"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="M 379.751,138.43701 H 505.04235 V 253.24016 H 379.751 Z"
id="path4109"
inkscape:connector-curvature="0"
style="fill:#00fdc8;fill-rule:evenodd" />
<path
d="M 379.751,138.43701 H 505.04235 V 253.24016 H 379.751 Z"
id="path4111"
inkscape:connector-curvature="0"
style="fill-rule:evenodd;stroke:#666666;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
<path
d="m 506.71874,190.09462 h 70.36065 v -6.68504 l 8.40525,13.37008 -8.40525,13.37007 v -6.68502 h -70.36065 z"
id="path4071-6"
inkscape:connector-curvature="0"
style="fill:#7c7ce0;fill-rule:evenodd;stroke-width:0.79288208" />
<g
aria-label="Some process"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:26.66666603px;line-height:1.25;font-family:'Comic Sans MS';-inkscape-font-specification:'Comic Sans MS';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none"
id="flowRoot4272">
<path
d="m 409.86816,182.0918 q -0.013,-0.54687 0.36459,-0.88542 0.39062,-0.35156 0.95052,-0.35156 0.63802,0 1.11979,0.72917 0.66406,0.98958 1.1849,1.34114 1.10677,0.72917 3.21614,0.72917 2.01823,0 3.75,-0.8724 2.21354,-1.11979 2.21354,-3.11197 0,-1.64063 -1.86198,-2.59115 -1.57552,-0.82031 -3.89322,-0.88542 -2.0573,-0.0781 -3.50261,-0.89843 -1.84896,-1.05469 -1.84896,-3.02084 0,-2.31771 2.63021,-4.23177 2.51302,-1.80989 5.19531,-1.80989 1.26302,0 2.85157,0.42968 2.10937,0.5599 2.10937,1.4323 0,1.14583 -1.17187,1.14583 -0.52084,0 -1.82292,-0.27344 -1.30208,-0.28646 -2.01823,-0.27344 -2.03125,0.026 -3.45052,1.00261 -1.45833,1.0026 -1.45833,2.43489 0,1.08073 1.92708,1.52344 0.71615,0.16927 2.77344,0.3125 2.98177,0.22136 4.66146,1.875 1.53645,1.51042 1.53645,3.6849 0,3.09895 -2.86458,4.94791 -2.48698,1.60157 -5.76823,1.60157 -2.38281,0 -4.36198,-0.91146 -2.42187,-1.11979 -2.46094,-3.07292 z"
style=""
id="path4896" />
<path
d="m 433.30566,186.10222 q -2.1875,0 -3.76302,-1.61459 -1.75781,-1.79687 -1.84896,-4.83072 -0.0781,-2.94271 1.48438,-5.1823 1.82292,-2.64323 5.07812,-2.64323 2.60417,0 3.94532,2.38282 1.0677,1.92708 1.02864,4.63541 -0.0391,2.89063 -1.51041,4.9349 -1.66667,2.31771 -4.41407,2.31771 z m 0.78125,-11.62761 q -1.91406,-0.013 -2.98177,1.82292 -0.84635,1.47135 -0.84635,3.35938 0,1.88802 1.02864,2.9427 0.85938,0.8724 2.01823,0.8724 1.35417,0 2.33073,-0.98958 1.10677,-1.09375 1.17188,-2.95573 0.18229,-5.03907 -2.72136,-5.05209 z"
style=""
id="path4898" />
<path
d="m 458.96973,186.41472 q -1.06771,0 -1.25,-1.13282 -0.23438,-1.43229 -0.46875,-4.03645 l -0.44271,-4.03646 q -0.13021,-0.89844 -0.44271,-1.75781 -0.44271,-1.17188 -0.96354,-1.17188 -0.48177,0 -1.79688,0.82031 -1.25,0.75521 -1.64062,1.21094 -0.013,1.22396 0.0651,2.44792 l 0.39063,3.47656 q 0.24739,2.05729 0.24739,3.39844 0,0.57291 -0.36458,0.9375 -0.35156,0.3776 -0.91146,0.3776 -0.55989,0 -0.92448,-0.3776 -0.35156,-0.36459 -0.35156,-0.9375 0,-1.57552 -0.32552,-4.64844 -0.3125,-3.08594 -0.3125,-4.66146 0,-1.38021 -0.39063,-2.14844 -0.91145,0.41667 -1.84895,1.35417 l -1.53646,1.71875 q -0.20834,0.3125 -0.66406,0.82031 0,0.78125 -0.14323,2.30469 -0.14323,1.52344 -0.14323,2.2526 0,0.39063 0.11718,1.09375 0.13021,0.70313 0.13021,1.01563 0,0.53385 -0.40364,0.89844 -0.39063,0.36458 -0.92448,0.36458 -1.47136,0 -1.47136,-3.34635 0,-0.78125 0.14323,-2.33073 0.14323,-1.54948 0.14323,-2.33073 0,-0.69011 -0.13021,-2.07031 -0.11718,-1.38021 -0.11718,-2.08334 0,-2.95573 1.32812,-2.95573 0.50781,0 0.92448,0.44271 0.42969,0.44271 0.42969,0.96354 0,0.2474 -0.0651,0.76823 -0.0651,0.52084 -0.0651,0.79427 l 0.026,1.36719 q 0.67708,-1.39323 1.99219,-2.51302 1.3151,-1.13281 2.23958,-1.13281 2.04427,0 2.8125,2.03125 0.71615,-0.95052 1.70573,-1.41927 0.98958,-0.48177 2.26563,-0.48177 2.17448,0 2.96875,2.34375 0.15625,0.46875 0.53385,3.65885 0.36458,2.23958 0.9375,7.46094 0.0391,0.52083 -0.35156,0.88541 -0.39063,0.36459 -0.95052,0.36459 z"
style=""
id="path4900" />
<path
d="m 469.24316,185.93295 q -2.94271,0 -4.76562,-1.47136 -1.99219,-1.6276 -1.99219,-4.55729 0,-3.38542 1.64062,-5.67708 1.78386,-2.5 4.76563,-2.5 2.16146,0 3.52865,0.72916 1.67968,0.89844 1.67968,2.76042 0,1.30208 -1.47135,2.36979 -0.65104,0.46875 -2.76042,1.45834 l -4.97396,2.35677 q 0.72917,1.08073 1.8099,1.6276 1.09375,0.54688 2.53906,0.54688 0.89844,0 2.10938,-0.33855 1.53646,-0.42968 1.97916,-1.11979 0.44271,-0.70312 0.95052,-0.70312 0.45573,0 0.8073,0.35156 0.36458,0.33854 0.36458,0.79427 0,1.44531 -2.33073,2.48698 -1.99219,0.88542 -3.88021,0.88542 z m -0.35156,-11.86198 q -1.58854,0 -2.64323,1.23698 -1.05469,1.22396 -1.57552,3.69791 l 3.9974,-1.90104 q 2.35677,-1.14583 3.35937,-1.99219 -1.11979,-1.04166 -3.13802,-1.04166 z"
style=""
id="path4902" />
<path
d="m 401.71712,219.12305 q -1.30208,0 -2.86458,-0.41667 l -0.026,6.22396 q 0,0.57291 -0.36458,0.9375 -0.35157,0.36458 -0.91146,0.36458 -0.5599,0 -0.92448,-0.36458 -0.35156,-0.36459 -0.35156,-0.9375 l 0.0391,-5.20834 v -5.19531 q -0.0391,-2.85156 -0.19531,-5.19531 l -0.19532,-2.64323 q 0,-0.92448 0.33854,-1.5625 0.39063,-0.76823 1.13282,-0.76823 0.50781,0 0.84635,0.36458 0.35156,0.35157 0.35156,0.8724 0,0.11719 -0.0391,0.42969 -0.0651,0.40364 -0.0911,0.74219 1.11979,-0.76823 2.16145,-1.14584 1.04167,-0.39062 1.99219,-0.39062 2.66927,0 3.90625,2.36979 0.97656,1.875 0.97656,4.96094 0,2.8125 -1.51041,4.63541 -1.60157,1.92709 -4.27084,1.92709 z m 0.89844,-11.28907 q -1.19792,0 -2.16146,0.53386 -0.6901,0.3776 -1.74479,1.38021 0.16927,2.95572 0.16927,6.21093 1.41927,0.54688 2.83854,0.54688 1.62761,0 2.48698,-1.21094 0.74219,-1.05469 0.74219,-2.73437 0,-2.3698 -0.58594,-3.54167 -0.57291,-1.1849 -1.74479,-1.1849 z"
style=""
id="path4904" />
<path
d="m 420.58431,209.04492 q -0.0912,1.6276 -1.30208,1.6276 -1.11979,0 -1.11979,-1.28906 0,-0.24739 -0.0391,-0.91146 l -0.0391,-1.09375 q -2.05729,0.2474 -3.29427,1.15886 -1.15885,0.84635 -1.92708,2.55208 l 0.026,7.13542 q 0,1.3151 -1.28906,1.3151 -1.14584,0 -1.14584,-1.3151 v -9.14063 q 0,-0.42968 0.026,-1.28906 0.0391,-0.85937 0.0391,-1.28906 0,-1.31511 1.14583,-1.31511 1.27604,0 1.28906,2.16146 2.5,-2.40885 5.54688,-2.40885 1.05468,0 1.57552,0.79427 0.52083,0.79427 0.52083,2.39583 0,0.67709 -0.013,0.91146 z"
style=""
id="path4906" />
<path
d="m 428.14941,219.43555 q -2.1875,0 -3.76302,-1.61459 -1.75781,-1.79687 -1.84896,-4.83073 -0.0781,-2.94271 1.48438,-5.18229 1.82292,-2.64323 5.07812,-2.64323 2.60417,0 3.94532,2.38282 1.06771,1.92708 1.02864,4.63541 -0.0391,2.89063 -1.51041,4.9349 -1.66667,2.31771 -4.41407,2.31771 z m 0.78125,-11.62761 q -1.91406,-0.013 -2.98177,1.82292 -0.84635,1.47135 -0.84635,3.35937 0,1.88802 1.02864,2.94271 0.85938,0.8724 2.01823,0.8724 1.35417,0 2.33073,-0.98959 1.10677,-1.09375 1.17188,-2.95573 0.18229,-5.03906 -2.72136,-5.05208 z"
style=""
id="path4908" />
<path
d="m 443.16244,219.48763 q -2.66927,0 -4.45313,-1.61458 -1.875,-1.69271 -1.875,-4.46615 0,-2.64323 1.75781,-5.45573 1.95313,-3.125 4.40105,-3.125 1.27604,0 2.85156,0.59896 1.99218,0.76823 1.99218,1.875 0,0.49479 -0.32552,0.88542 -0.32552,0.3776 -0.80729,0.3776 -0.3776,0 -0.66406,-0.22135 -0.27344,-0.23438 -0.53385,-0.45573 -0.76823,-0.61198 -2.51302,-0.61198 -1.32813,0 -2.63021,2.36979 -1.1849,2.17448 -1.1849,3.76302 0,1.71875 1.17188,2.70833 1.09375,0.92448 2.8125,0.92448 0.78125,0 1.66666,-0.41666 l 1.51042,-0.8073 q 0.39062,-0.20833 0.55989,-0.20833 0.48178,0 0.83334,0.39063 0.35156,0.3776 0.35156,0.87239 0,0.95052 -2.00521,1.83594 -1.75781,0.78125 -2.91666,0.78125 z"
style=""
id="path4910" />
<path
d="m 457.05566,219.26627 q -2.94271,0 -4.76562,-1.47135 -1.99219,-1.6276 -1.99219,-4.55729 0,-3.38542 1.64063,-5.67708 1.78385,-2.5 4.76562,-2.5 2.16146,0 3.52865,0.72916 1.67968,0.89844 1.67968,2.76042 0,1.30208 -1.47135,2.36979 -0.65104,0.46875 -2.76042,1.45833 l -4.97396,2.35677 q 0.72917,1.08073 1.8099,1.62761 1.09375,0.54687 2.53906,0.54687 0.89844,0 2.10938,-0.33854 1.53646,-0.42969 1.97916,-1.11979 0.44271,-0.70312 0.95052,-0.70312 0.45573,0 0.8073,0.35156 0.36458,0.33854 0.36458,0.79427 0,1.44531 -2.33073,2.48698 -1.99219,0.88541 -3.88021,0.88541 z M 456.7041,207.4043 q -1.58854,0 -2.64323,1.23698 -1.05469,1.22395 -1.57552,3.69791 l 3.9974,-1.90104 q 2.35677,-1.14583 3.35937,-1.99219 -1.11979,-1.04166 -3.13802,-1.04166 z"
style=""
id="path4912" />
<path
d="m 474.39941,208.64128 q -0.71615,0 -1.0026,-0.48178 -0.19532,-0.33854 -0.33855,-1.40625 -0.97656,0.2474 -3.00781,0.91146 -2.09635,0.83334 -2.10937,2.05729 0.39062,0.14323 0.76823,0.20834 2.99479,0.53385 4.45312,1.47135 2.1224,1.36719 2.1224,3.95833 0,2.1224 -1.83594,3.20313 -1.52344,0.88542 -3.85417,0.88542 -1.70573,0 -3.25521,-0.57292 -2.0052,-0.72917 -2.0052,-2.03125 0,-0.52083 0.40364,-0.91146 0.41667,-0.39062 0.9375,-0.39062 0.40365,0 1.04167,0.625 0.3125,0.29948 1.40625,0.49479 0.87239,0.15625 1.47135,0.15625 1.09375,0 1.92709,-0.26042 1.21093,-0.3776 1.21093,-1.19792 0,-1.77083 -2.47396,-2.39583 l -1.35416,-0.29948 q -1.73177,-0.3776 -2.46094,-0.88541 -1.05469,-0.72917 -1.05469,-2.20053 0,-2.72135 2.56511,-3.8802 0.78125,-0.35157 2.5,-0.83334 1.71875,-0.49479 2.46093,-0.83333 0.53386,-0.23437 1.15886,-0.23437 0.55989,0 0.91146,0.36458 0.36458,0.36458 0.36458,0.95052 0,0.3776 0.15625,1.10677 0.16927,0.72917 0.16927,1.10677 0,0.58594 -0.36458,0.95052 -0.35156,0.36459 -0.91146,0.36459 z"
style=""
id="path4914" />
<path
d="m 487.36816,208.64128 q -0.71615,0 -1.0026,-0.48178 -0.19532,-0.33854 -0.33855,-1.40625 -0.97656,0.2474 -3.00781,0.91146 -2.09635,0.83334 -2.10937,2.05729 0.39062,0.14323 0.76823,0.20834 2.99479,0.53385 4.45312,1.47135 2.1224,1.36719 2.1224,3.95833 0,2.1224 -1.83594,3.20313 -1.52344,0.88542 -3.85417,0.88542 -1.70573,0 -3.25521,-0.57292 -2.0052,-0.72917 -2.0052,-2.03125 0,-0.52083 0.40364,-0.91146 0.41667,-0.39062 0.9375,-0.39062 0.40365,0 1.04167,0.625 0.3125,0.29948 1.40625,0.49479 0.87239,0.15625 1.47135,0.15625 1.09375,0 1.92709,-0.26042 1.21093,-0.3776 1.21093,-1.19792 0,-1.77083 -2.47396,-2.39583 l -1.35416,-0.29948 q -1.73177,-0.3776 -2.46094,-0.88541 -1.05469,-0.72917 -1.05469,-2.20053 0,-2.72135 2.56511,-3.8802 0.78125,-0.35157 2.5,-0.83334 1.71875,-0.49479 2.46093,-0.83333 0.53386,-0.23437 1.15886,-0.23437 0.55989,0 0.91146,0.36458 0.36458,0.36458 0.36458,0.95052 0,0.3776 0.15625,1.10677 0.16927,0.72917 0.16927,1.10677 0,0.58594 -0.36458,0.95052 -0.35156,0.36459 -0.91146,0.36459 z"
style=""
id="path4916" />
</g>
<rect
style="fill:#ffffff"
id="rect148"
width="28.983051"
height="45.762711"
x="212.03391"
y="388.22034" />
</g>
</svg>
</center></p>
<p>This process can be <em>anything</em>. For example, it can be:</p>
<ul>
<li>a process calculating some statistics ("tell me how many users have red hair")</li>
<li>a de-identification strategy ("remove names and last three digits of ZIP
codes")</li>
<li>a machine learning training process ("build a model to predict which users
like cats")</li>
<li>… you get the idea.</li>
</ul>
<p>To make a process differentially private, you usually have to modify it a little
bit. Typically, you add some randomness, or <em>noise</em>, in some places. What
exactly you do, and how much noise you add, depends on which process you're
modifying. I'll abstract that part away and simply say that your process is now
doing some unspecified ✨ magic ✨.</p>
<p><center>
<svg
role="img"
aria-label="Same diagram as before, except the arrow is labeled with a silly "magic" gif."
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
version="1.1"
viewBox="0 0 699.45947 142.76904"
stroke-miterlimit="10"
id="svg4123"
sodipodi:docname="magic.svg"
inkscape:version="0.92.3 (2405546, 2018-03-11)"
width="699.45947"
height="142.76904"
style="fill:none;stroke:none;stroke-linecap:square;stroke-miterlimit:10">
<title>Same diagram as before, except the arrow is labeled with a silly "magic" gif.</title>
<desc>Same diagram as before, except the arrow is labeled with a silly "magic" gif.</desc>
<metadata
id="metadata4129">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
<dc:title />
</cc:Work>
</rdf:RDF>
</metadata>
<defs
id="defs4127">
<clipPath
id="p.3-4">
<path
d="M 0,0 H 275 V 252 H 0 Z"
id="path4077-0"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
</defs>
<sodipodi:namedview
pagecolor="#ffffff"
bordercolor="#666666"
borderopacity="1"
objecttolerance="10"
gridtolerance="10"
guidetolerance="10"
inkscape:pageopacity="0"
inkscape:pageshadow="2"
inkscape:window-width="2556"
inkscape:window-height="1436"
id="namedview4125"
showgrid="false"
inkscape:zoom="1.3111111"
inkscape:cx="608.41836"
inkscape:cy="-64.033451"
inkscape:window-x="0"
inkscape:window-y="1200"
inkscape:window-maximized="0"
inkscape:current-layer="g4121"
fit-margin-top="0"
fit-margin-left="0"
fit-margin-right="0"
fit-margin-bottom="0"
viewbox-x="82"
viewbox-width="650" />
<clipPath
id="p.0">
<path
d="M 0,0 H 960 V 720 H 0 Z"
id="path4036"
inkscape:connector-curvature="0"
style="clip-rule:nonzero" />
</clipPath>
<g
clip-path="url(#p.0)"
id="g4121"
transform="translate(-76.08921,-124.34908)">
<path
d="M 0,0 H 960 V 720 H 0 Z"
id="path4039"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 76.08921,124.34908 h 223.2992 V 267.11813 H 76.08921 Z"
id="path4041"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<g
transform="matrix(0.22307612,0,0,0.22307664,76.089214,124.34908)"
id="g4048">
<clipPath
id="p.1">
<path
d="M 0,0 H 1001 V 640 H 0 Z"
id="path4043"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
<image
clip-path="url(#p.1)"
width="1001"
height="640"
x="0"
y="0"
preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/quiestce.jpeg"
id="image4046"
style="fill:#000000" />
</g>
<path
d="M 76.086586,339.67847 H 299.38579 V 482.44751 H 76.086586 Z"
id="path4050"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 604.81974,178.70273 v 0 c -1.51233,-10.41394 3.45307,-20.72311 12.78925,-26.55296 9.33618,-5.82984 21.40576,-6.15791 31.08722,-0.84502 v 0 c 3.42944,-6.0551 9.70636,-10.23573 16.93207,-11.27733 7.2257,-1.04159 14.55151,1.17818 19.76159,5.98789 v 0 c 2.92145,-5.48995 8.65778,-9.17848 15.1734,-9.75675 6.51569,-0.57824 12.88855,2.03563 16.85718,6.9141 v 0 c 5.27802,-5.81934 13.67554,-8.2695 21.55884,-6.29033 7.8833,1.97917 13.83649,8.03222 15.28363,15.53996 v 0 c 6.46649,1.65271 11.8529,5.85413 14.76764,11.51872 2.91473,5.66462 3.07178,12.23673 0.43066,18.01831 v 0 c 6.36756,7.76532 7.85712,18.11251 3.91272,27.18018 -3.9444,9.06767 -12.7301,15.49351 -23.07849,16.87953 -0.0729,8.51033 -5.05414,16.31931 -13.02356,20.41701 -7.96942,4.09772 -17.68268,3.84427 -25.39587,-0.66263 -3.2854,10.1926 -12.53272,17.6922 -23.74671,19.25873 -11.21405,1.56653 -22.38446,-3.08087 -28.68524,-11.93434 -7.72339,4.3639 -16.99084,5.621 -25.71185,3.48773 -8.72095,-2.13329 -16.16077,-7.47725 -20.64124,-14.8264 v 0 c -7.89233,0.86539 -15.52319,-2.96604 -19.10534,-9.59276 -3.58216,-6.62671 -2.35303,-14.63803 3.07733,-20.05798 v 0 c -7.04022,-3.8826 -10.63251,-11.58696 -8.90375,-19.09555 1.72882,-7.50861 8.38696,-13.11998 16.50256,-13.90799 z"
id="path4059"
inkscape:connector-curvature="0"
style="fill:#f6cd4c;fill-rule:evenodd" />
<path
d="m 597.06497,212.10764 v 0 c 3.32227,1.83222 7.16028,2.66335 10.99872,2.38181 m 5.02759,27.2696 c 1.65063,-0.18098 3.26861,-0.56427 4.81219,-1.13993 m 41.53858,12.47812 c -1.16095,-1.63134 -2.133,-3.37456 -2.89954,-5.19996 m 55.3335,-2.12474 v 0 c 0.59893,-1.85826 0.98706,-3.77081 1.15777,-5.70575 m 37.26019,-14.04791 c 0.0776,-9.06056 -5.41461,-17.3565 -14.11767,-21.32448 m 33.2832,-22.73425 c -1.40942,3.08528 -3.56109,5.8222 -6.28625,7.99617 m -8.91071,-37.53403 v 0 c 0.24011,1.24591 0.35126,2.51055 0.33185,3.77639 m -37.17371,-13.02565 v 0 c -1.31659,1.45164 -2.4013,3.07382 -3.22033,4.81604 m -28.81055,-1.97402 v 0 c -0.70166,1.31851 -1.22559,2.71372 -1.55975,4.15347 m -35.13446,1.13608 v 0 c 2.04858,1.12421 3.94384,2.47731 5.64404,4.02959 m -49.5199,23.36855 v 0 c 0.20844,1.4353 0.53784,2.85292 0.98499,4.23957"
id="path4061"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 636.30257,211.8428 h -0.1875 q -0.78125,0 -1.25,-0.48437 -0.46875,-0.48438 -0.46875,-1.17188 0,-0.46875 0.5,-2.92187 l 1.375,-7.10938 q 0.46875,-2.42187 2.0625,-13.4375 l 0.39063,-2.70312 q 0.14062,-1.01563 1.10937,-1.84375 0.96875,-0.84375 1.78125,-0.84375 0.4375,0 2.9375,1.3125 2.70313,1.45312 3.20313,1.59375 5.09375,1.875 8.375,5.6875 3.29687,3.8125 3.29687,8.23437 0,4.04688 -2.40625,7.92188 -2.39062,3.875 -5.78125,5.71875 -3.39062,1.84375 -8.67187,1.84375 -1.53125,0 -3.46875,-0.53125 -1.9375,-0.51563 -2.79688,-1.26563 z m 6,-25.40625 -1.64062,11.20313 -2,10.67187 q 0.0625,0.0469 0.14062,0.0781 1.84375,1.32812 4.375,1.32812 5.01563,0 7.375,-1.79687 2.35938,-1.79688 3.60938,-4.21875 1.25,-2.42188 1.25,-4.92188 0,-2.51562 -1.40625,-4.64062 -1.39063,-2.125 -3.76563,-3.57813 -2.35937,-1.46875 -7.9375,-4.125 z m 32.16406,24.28125 q -4.21875,2.03125 -6.01562,2.03125 -7.29688,0 -7.29688,-6.95312 0,-6.46875 4.07813,-10.46875 4.09375,-4 8.98437,-4 1.89063,0 3.90625,0.96875 2.03125,0.96875 2.03125,2.25 0,0.79687 -0.53125,1.25 -0.39062,0.76562 -1.10937,4.40625 -0.70313,3.64062 -0.70313,5.25 0,1.96875 1.01563,5.42187 l 0.0781,0.3125 q -0.4375,1.82813 -2.17187,1.82813 -0.29688,0 -1.09375,-0.79688 -0.79688,-0.79687 -1.17188,-1.5 z m 1.42188,-15.1875 q -1.21875,-0.73437 -1.95313,-0.73437 -3.25,0 -6.07812,2.95312 -2.82813,2.9375 -2.82813,7.3125 0,4.29688 3.71875,4.29688 2.90625,0 5.46875,-2.15625 0.3125,-8.1875 1.67188,-11.67188 z m 16.45312,-0.34375 q -0.23437,0.9375 -0.60937,2.1875 -1.64063,5.42188 -1.64063,8.26563 0,3.35937 1.5625,3.35937 1.54688,0 2.28125,-0.95312 1.01563,-1.32813 1.875,-1.32813 0.65625,0 1.25,0.48438 0.59375,0.48437 0.59375,1.26562 0,1.71875 -2.15625,2.95313 -2.15625,1.21875 -4.29687,1.21875 -5.03125,0 -5.03125,-6.28125 0,-3.4375 1.54687,-8.39063 0.40625,-1.3125 0.78125,-2.625 -4.25,-0.20312 -4.85937,-0.70312 -0.60938,-0.5 -0.60938,-1.5625 0,-0.85938 0.57813,-1.42188 0.59375,-0.57812 1.60937,-0.57812 l 4.03125,0.32812 q 0.125,-0.51562 0.1875,-1.17187 0.0312,-0.67188 0.125,-1.48438 l 0.34375,-2.42187 q 0.10938,-0.79688 0.71875,-1.34375 0.625,-0.54688 1.46875,-0.54688 1.64063,0 1.64063,2.625 0,0.76563 -0.23438,1.98438 l -0.39062,2.35937 q 2.60937,-0.32812 2.78125,-0.32812 2.03125,0 2.79687,0.40625 0.76563,0.39062 0.76563,1.46875 0,0.95312 -0.65625,1.54687 -0.65625,0.57813 -1.57813,0.57813 l -1.92187,-0.0781 q -1.45313,0 -2.95313,0.1875 z m 21.44141,15.53125 q -4.21875,2.03125 -6.01563,2.03125 -7.29687,0 -7.29687,-6.95312 0,-6.46875 4.07812,-10.46875 4.09375,-4 8.98438,-4 1.89062,0 3.90625,0.96875 2.03125,0.96875 2.03125,2.25 0,0.79687 -0.53125,1.25 -0.39063,0.76562 -1.10938,4.40625 -0.70312,3.64062 -0.70312,5.25 0,1.96875 1.01562,5.42187 l 0.0781,0.3125 q -0.4375,1.82813 -2.17188,1.82813 -0.29687,0 -1.09375,-0.79688 -0.79687,-0.79687 -1.17187,-1.5 z m 1.42187,-15.1875 q -1.21875,-0.73437 -1.95312,-0.73437 -3.25,0 -6.07813,2.95312 -2.82812,2.9375 -2.82812,7.3125 0,4.29688 3.71875,4.29688 2.90625,0 5.46875,-2.15625 0.3125,-8.1875 1.67187,-11.67188 z"
id="path4063"
inkscape:connector-curvature="0"
style="fill:#000000;fill-rule:nonzero" />
<path
d="m 597.06237,427.43705 v 0 c 3.32227,1.83218 7.16028,2.66333 10.99872,2.38181 m 5.02759,27.26959 c 1.65063,-0.181 3.26861,-0.56427 4.81219,-1.13992 m 41.53858,12.47811 c -1.16095,-1.63134 -2.133,-3.37457 -2.89954,-5.19998 m 55.3335,-2.12472 v 0 c 0.59893,-1.85825 0.98706,-3.77082 1.15777,-5.70575 m 37.26019,-14.04792 c 0.0776,-9.06054 -5.41461,-17.3565 -14.11767,-21.32449 m 33.2832,-22.73425 c -1.40942,3.0853 -3.56109,5.82223 -6.28625,7.99618 m -8.91071,-37.53402 v 0 c 0.24011,1.24591 0.35126,2.51056 0.33185,3.77639 m -37.17371,-13.02566 v 0 c -1.31659,1.45163 -2.4013,3.07382 -3.22033,4.81604 m -28.81055,-1.974 v 0 c -0.70166,1.31848 -1.22559,2.71372 -1.55975,4.15347 m -35.13446,1.13608 v 0 c 2.04858,1.12421 3.94384,2.4773 5.64404,4.02957 m -49.5199,23.36856 v 0 c 0.20844,1.43531 0.53784,2.85291 0.98499,4.23957"
id="path4067"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 299.38846,189.04987 h 70.36065 v -6.68504 l 8.40525,13.37008 -8.40525,13.37007 v -6.68502 h -70.36065 z"
id="path4071"
inkscape:connector-curvature="0"
style="fill:#7c7ce0;fill-rule:evenodd;stroke-width:0.79288208" />
<path
d="M 427.88773,353.66272 H 553.16596 V 468.46329 H 427.88773 Z"
id="path4075"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="M 427.809,138.3307 H 553.08723 V 253.13125 H 427.809 Z"
id="path4100"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="M 379.751,138.43701 H 505.04235 V 253.24016 H 379.751 Z"
id="path4109"
inkscape:connector-curvature="0"
style="fill:#00fdc8;fill-rule:evenodd" />
<path
d="M 379.751,138.43701 H 505.04235 V 253.24016 H 379.751 Z"
id="path4111"
inkscape:connector-curvature="0"
style="fill-rule:evenodd;stroke:#666666;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
<path
d="m 506.71874,190.09462 h 70.36065 v -6.68504 l 8.40525,13.37008 -8.40525,13.37007 v -6.68502 h -70.36065 z"
id="path4071-6"
inkscape:connector-curvature="0"
style="fill:#7c7ce0;fill-rule:evenodd;stroke-width:0.79288208" />
<flowRoot
xml:space="preserve"
id="flowRoot4272"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:26.66666603px;line-height:1.25;font-family:'Comic Sans MS';-inkscape-font-specification:'Comic Sans MS';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none"><flowRegion
id="flowRegion4274"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:26.66666603px;font-family:'Comic Sans MS';-inkscape-font-specification:'Comic Sans MS';text-align:center;text-anchor:middle"><rect
id="rect4276"
width="125.97456"
height="90.782555"
x="379.06781"
y="162.45763"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:26.66666603px;font-family:'Comic Sans MS';-inkscape-font-specification:'Comic Sans MS';text-align:center;text-anchor:middle" /></flowRegion><flowPara
id="flowPara4278">Some process</flowPara></flowRoot> <g
transform="matrix(0.45555722,0,0,0.45555774,379.751,138.43701)"
id="g4082-1"
inkscape:transform-center-x="-34.322034"
inkscape:transform-center-y="102.9661">
<clipPath
id="clipPath4292">
<path
d="M 0,0 H 275 V 252 H 0 Z"
id="path4290"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
<image
clip-path="url(#p.3-4)"
width="275"
height="252"
x="0"
y="0"
preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/magic.gif"
id="image4080-3"
style="fill:#000000" />
</g>
<rect
style="fill:#ffffff"
id="rect148"
width="28.983051"
height="45.762711"
x="212.03391"
y="388.22034" />
</g>
</svg>
</center></p>
<p>Now, remove somebody from your database, and run your new process on it. If the
new process is differentially private, then the two outputs are <em>basically the
same</em>. This must be true no matter who you remove, and what database you had in
the first place.</p>
<p><center>
<svg
role="img"
aria-label="Same diagram as before, duplicated, with the bottom line missing one person in the database. A double arrow labeled "basically the same" points to the two outputs."
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
version="1.1"
viewBox="0 0 734.41882 358.09845"
stroke-miterlimit="10"
id="svg4123"
sodipodi:docname="full-diagram.svg"
inkscape:version="0.92.2pre0 (973e216, 2017-07-25)"
width="734.41882"
height="358.09845"
style="fill:none;stroke:none;stroke-linecap:square;stroke-miterlimit:10">
<title>Same diagram as before, duplicated, with the bottom line missing one person in the database. A double arrow labeled "basically the same" points to the two outputs.</title>
<desc>Same diagram as before, duplicated, with the bottom line missing one person in the database. A double arrow labeled "basically the same" points to the two outputs.</desc>
<metadata
id="metadata4129">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
<dc:title></dc:title>
</cc:Work>
</rdf:RDF>
</metadata>
<defs
id="defs4127">
<clipPath
id="p.3-4">
<path
d="M 0,0 H 275 V 252 H 0 Z"
id="path4077-0"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
</defs>
<sodipodi:namedview
pagecolor="#ffffff"
bordercolor="#666666"
borderopacity="1"
objecttolerance="10"
gridtolerance="10"
guidetolerance="10"
inkscape:pageopacity="0"
inkscape:pageshadow="2"
inkscape:window-width="1918"
inkscape:window-height="1078"
id="namedview4125"
showgrid="false"
inkscape:zoom="1.3111111"
inkscape:cx="532.33439"
inkscape:cy="127.30279"
inkscape:window-x="0"
inkscape:window-y="0"
inkscape:window-maximized="0"
inkscape:current-layer="g4121"
fit-margin-top="0"
fit-margin-left="0"
fit-margin-right="0"
fit-margin-bottom="0"
viewbox-x="82"
viewbox-width="650" />
<clipPath
id="p.0">
<path
d="M 0,0 H 960 V 720 H 0 Z"
id="path4036"
inkscape:connector-curvature="0"
style="clip-rule:nonzero" />
</clipPath>
<g
clip-path="url(#p.0)"
id="g4121"
transform="translate(-76.086586,-124.34907)">
<path
d="M 0,0 H 960 V 720 H 0 Z"
id="path4039"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 76.08921,124.34908 h 223.2992 V 267.11813 H 76.08921 Z"
id="path4041"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<g
transform="matrix(0.22307612,0,0,0.22307664,76.089214,124.34908)"
id="g4048">
<clipPath
id="p.1">
<path
d="M 0,0 H 1001 V 640 H 0 Z"
id="path4043"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
<image
clip-path="url(#p.1)"
width="1001"
height="640"
x="0"
y="0"
preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/quiestce.jpeg"
id="image4046"
style="fill:#000000" />
</g>
<path
d="M 76.086586,339.67847 H 299.38579 V 482.44751 H 76.086586 Z"
id="path4050"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<g
transform="matrix(0.22307612,0,0,0.22307664,76.086589,339.67847)"
id="g4057">
<clipPath
id="p.2">
<path
d="M 0,0 H 1001 V 640 H 0 Z"
id="path4052"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
<image
clip-path="url(#p.2)"
width="1001"
height="640"
x="0"
y="0"
preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/quiestce.jpeg"
id="image4055"
style="fill:#000000" />
</g>
<path
d="m 604.81974,178.70273 v 0 c -1.51233,-10.41394 3.45307,-20.72311 12.78925,-26.55296 9.33618,-5.82984 21.40576,-6.15791 31.08722,-0.84502 v 0 c 3.42944,-6.0551 9.70636,-10.23573 16.93207,-11.27733 7.2257,-1.04159 14.55151,1.17818 19.76159,5.98789 v 0 c 2.92145,-5.48995 8.65778,-9.17848 15.1734,-9.75675 6.51569,-0.57824 12.88855,2.03563 16.85718,6.9141 v 0 c 5.27802,-5.81934 13.67554,-8.2695 21.55884,-6.29033 7.8833,1.97917 13.83649,8.03222 15.28363,15.53996 v 0 c 6.46649,1.65271 11.8529,5.85413 14.76764,11.51872 2.91473,5.66462 3.07178,12.23673 0.43066,18.01831 v 0 c 6.36756,7.76532 7.85712,18.11251 3.91272,27.18018 -3.9444,9.06767 -12.7301,15.49351 -23.07849,16.87953 -0.0729,8.51033 -5.05414,16.31931 -13.02356,20.41701 -7.96942,4.09772 -17.68268,3.84427 -25.39587,-0.66263 -3.2854,10.1926 -12.53272,17.6922 -23.74671,19.25873 -11.21405,1.56653 -22.38446,-3.08087 -28.68524,-11.93434 -7.72339,4.3639 -16.99084,5.621 -25.71185,3.48773 -8.72095,-2.13329 -16.16077,-7.47725 -20.64124,-14.8264 v 0 c -7.89233,0.86539 -15.52319,-2.96604 -19.10534,-9.59276 -3.58216,-6.62671 -2.35303,-14.63803 3.07733,-20.05798 v 0 c -7.04022,-3.8826 -10.63251,-11.58696 -8.90375,-19.09555 1.72882,-7.50861 8.38696,-13.11998 16.50256,-13.90799 z"
id="path4059"
inkscape:connector-curvature="0"
style="fill:#f6cd4c;fill-rule:evenodd" />
<path
d="m 597.06497,212.10764 v 0 c 3.32227,1.83222 7.16028,2.66335 10.99872,2.38181 m 5.02759,27.2696 c 1.65063,-0.18098 3.26861,-0.56427 4.81219,-1.13993 m 41.53858,12.47812 c -1.16095,-1.63134 -2.133,-3.37456 -2.89954,-5.19996 m 55.3335,-2.12474 v 0 c 0.59893,-1.85826 0.98706,-3.77081 1.15777,-5.70575 m 37.26019,-14.04791 c 0.0776,-9.06056 -5.41461,-17.3565 -14.11767,-21.32448 m 33.2832,-22.73425 c -1.40942,3.08528 -3.56109,5.8222 -6.28625,7.99617 m -8.91071,-37.53403 v 0 c 0.24011,1.24591 0.35126,2.51055 0.33185,3.77639 m -37.17371,-13.02565 v 0 c -1.31659,1.45164 -2.4013,3.07382 -3.22033,4.81604 m -28.81055,-1.97402 v 0 c -0.70166,1.31851 -1.22559,2.71372 -1.55975,4.15347 m -35.13446,1.13608 v 0 c 2.04858,1.12421 3.94384,2.47731 5.64404,4.02959 m -49.5199,23.36855 v 0 c 0.20844,1.4353 0.53784,2.85292 0.98499,4.23957"
id="path4061"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 636.30257,211.8428 h -0.1875 q -0.78125,0 -1.25,-0.48437 -0.46875,-0.48438 -0.46875,-1.17188 0,-0.46875 0.5,-2.92187 l 1.375,-7.10938 q 0.46875,-2.42187 2.0625,-13.4375 l 0.39063,-2.70312 q 0.14062,-1.01563 1.10937,-1.84375 0.96875,-0.84375 1.78125,-0.84375 0.4375,0 2.9375,1.3125 2.70313,1.45312 3.20313,1.59375 5.09375,1.875 8.375,5.6875 3.29687,3.8125 3.29687,8.23437 0,4.04688 -2.40625,7.92188 -2.39062,3.875 -5.78125,5.71875 -3.39062,1.84375 -8.67187,1.84375 -1.53125,0 -3.46875,-0.53125 -1.9375,-0.51563 -2.79688,-1.26563 z m 6,-25.40625 -1.64062,11.20313 -2,10.67187 q 0.0625,0.0469 0.14062,0.0781 1.84375,1.32812 4.375,1.32812 5.01563,0 7.375,-1.79687 2.35938,-1.79688 3.60938,-4.21875 1.25,-2.42188 1.25,-4.92188 0,-2.51562 -1.40625,-4.64062 -1.39063,-2.125 -3.76563,-3.57813 -2.35937,-1.46875 -7.9375,-4.125 z m 32.16406,24.28125 q -4.21875,2.03125 -6.01562,2.03125 -7.29688,0 -7.29688,-6.95312 0,-6.46875 4.07813,-10.46875 4.09375,-4 8.98437,-4 1.89063,0 3.90625,0.96875 2.03125,0.96875 2.03125,2.25 0,0.79687 -0.53125,1.25 -0.39062,0.76562 -1.10937,4.40625 -0.70313,3.64062 -0.70313,5.25 0,1.96875 1.01563,5.42187 l 0.0781,0.3125 q -0.4375,1.82813 -2.17187,1.82813 -0.29688,0 -1.09375,-0.79688 -0.79688,-0.79687 -1.17188,-1.5 z m 1.42188,-15.1875 q -1.21875,-0.73437 -1.95313,-0.73437 -3.25,0 -6.07812,2.95312 -2.82813,2.9375 -2.82813,7.3125 0,4.29688 3.71875,4.29688 2.90625,0 5.46875,-2.15625 0.3125,-8.1875 1.67188,-11.67188 z m 16.45312,-0.34375 q -0.23437,0.9375 -0.60937,2.1875 -1.64063,5.42188 -1.64063,8.26563 0,3.35937 1.5625,3.35937 1.54688,0 2.28125,-0.95312 1.01563,-1.32813 1.875,-1.32813 0.65625,0 1.25,0.48438 0.59375,0.48437 0.59375,1.26562 0,1.71875 -2.15625,2.95313 -2.15625,1.21875 -4.29687,1.21875 -5.03125,0 -5.03125,-6.28125 0,-3.4375 1.54687,-8.39063 0.40625,-1.3125 0.78125,-2.625 -4.25,-0.20312 -4.85937,-0.70312 -0.60938,-0.5 -0.60938,-1.5625 0,-0.85938 0.57813,-1.42188 0.59375,-0.57812 1.60937,-0.57812 l 4.03125,0.32812 q 0.125,-0.51562 0.1875,-1.17187 0.0312,-0.67188 0.125,-1.48438 l 0.34375,-2.42187 q 0.10938,-0.79688 0.71875,-1.34375 0.625,-0.54688 1.46875,-0.54688 1.64063,0 1.64063,2.625 0,0.76563 -0.23438,1.98438 l -0.39062,2.35937 q 2.60937,-0.32812 2.78125,-0.32812 2.03125,0 2.79687,0.40625 0.76563,0.39062 0.76563,1.46875 0,0.95312 -0.65625,1.54687 -0.65625,0.57813 -1.57813,0.57813 l -1.92187,-0.0781 q -1.45313,0 -2.95313,0.1875 z m 21.44141,15.53125 q -4.21875,2.03125 -6.01563,2.03125 -7.29687,0 -7.29687,-6.95312 0,-6.46875 4.07812,-10.46875 4.09375,-4 8.98438,-4 1.89062,0 3.90625,0.96875 2.03125,0.96875 2.03125,2.25 0,0.79687 -0.53125,1.25 -0.39063,0.76562 -1.10938,4.40625 -0.70312,3.64062 -0.70312,5.25 0,1.96875 1.01562,5.42187 l 0.0781,0.3125 q -0.4375,1.82813 -2.17188,1.82813 -0.29687,0 -1.09375,-0.79688 -0.79687,-0.79687 -1.17187,-1.5 z m 1.42187,-15.1875 q -1.21875,-0.73437 -1.95312,-0.73437 -3.25,0 -6.07813,2.95312 -2.82812,2.9375 -2.82812,7.3125 0,4.29688 3.71875,4.29688 2.90625,0 5.46875,-2.15625 0.3125,-8.1875 1.67187,-11.67188 z"
id="path4063"
inkscape:connector-curvature="0"
style="fill:#000000;fill-rule:nonzero" />
<path
d="m 604.81711,394.03214 v 0 c -1.51233,-10.41397 3.45307,-20.72311 12.78925,-26.55298 9.33618,-5.82983 21.40576,-6.1579 31.08722,-0.845 v 0 c 3.42944,-6.05511 9.70636,-10.23575 16.93207,-11.27734 7.2257,-1.0416 14.55151,1.17819 19.76159,5.98788 v 0 c 2.92145,-5.48993 8.65778,-9.17847 15.1734,-9.75674 6.51569,-0.57825 12.88855,2.03564 16.85718,6.91409 v 0 c 5.27802,-5.81934 13.67554,-8.2695 21.55884,-6.29031 7.8833,1.97915 13.83649,8.03222 15.28363,15.53994 v 0 c 6.46649,1.65271 11.8529,5.85413 14.76764,11.51874 2.91473,5.66461 3.07178,12.23673 0.43066,18.01831 v 0 c 6.36756,7.76532 7.85712,18.11249 3.91272,27.18018 -3.9444,9.06766 -12.7301,15.4935 -23.07849,16.87952 -0.0729,8.51034 -5.05414,16.3193 -13.02356,20.41702 -7.96942,4.09769 -17.68268,3.84427 -25.39587,-0.66266 -3.2854,10.19263 -12.53272,17.69223 -23.74671,19.25876 -11.21405,1.56653 -22.38446,-3.08087 -28.68524,-11.93436 -7.72339,4.36392 -16.99084,5.621 -25.71185,3.48773 -8.72095,-2.13327 -16.16077,-7.47723 -20.64124,-14.82638 v 0 c -7.89233,0.86539 -15.52319,-2.96607 -19.10534,-9.59277 -3.58216,-6.62671 -2.35303,-14.63804 3.07733,-20.05799 v 0 c -7.04022,-3.8826 -10.63257,-11.58694 -8.90375,-19.09555 1.72882,-7.5086 8.38696,-13.11996 16.50256,-13.90799 z"
id="path4065"
inkscape:connector-curvature="0"
style="fill:#f6cd4c;fill-rule:evenodd" />
<path
d="m 597.06237,427.43705 v 0 c 3.32227,1.83218 7.16028,2.66333 10.99872,2.38181 m 5.02759,27.26959 c 1.65063,-0.181 3.26861,-0.56427 4.81219,-1.13992 m 41.53858,12.47811 c -1.16095,-1.63134 -2.133,-3.37457 -2.89954,-5.19998 m 55.3335,-2.12472 v 0 c 0.59893,-1.85825 0.98706,-3.77082 1.15777,-5.70575 m 37.26019,-14.04792 c 0.0776,-9.06054 -5.41461,-17.3565 -14.11767,-21.32449 m 33.2832,-22.73425 c -1.40942,3.0853 -3.56109,5.82223 -6.28625,7.99618 m -8.91071,-37.53402 v 0 c 0.24011,1.24591 0.35126,2.51056 0.33185,3.77639 m -37.17371,-13.02566 v 0 c -1.31659,1.45163 -2.4013,3.07382 -3.22033,4.81604 m -28.81055,-1.974 v 0 c -0.70166,1.31848 -1.22559,2.71372 -1.55975,4.15347 m -35.13446,1.13608 v 0 c 2.04858,1.12421 3.94384,2.4773 5.64404,4.02957 m -49.5199,23.36856 v 0 c 0.20844,1.43531 0.53784,2.85291 0.98499,4.23957"
id="path4067"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="m 636.29997,427.17219 h -0.1875 q -0.78125,0 -1.25,-0.48437 -0.46875,-0.48438 -0.46875,-1.17188 0,-0.46875 0.5,-2.92187 l 1.375,-7.10938 q 0.46875,-2.42187 2.0625,-13.4375 l 0.39063,-2.70312 q 0.14062,-1.01563 1.10937,-1.84375 0.96875,-0.84375 1.78125,-0.84375 0.4375,0 2.9375,1.3125 2.70313,1.45312 3.20313,1.59375 5.09375,1.875 8.375,5.6875 3.29687,3.8125 3.29687,8.23437 0,4.04688 -2.40625,7.92188 -2.39062,3.875 -5.78125,5.71875 -3.39062,1.84375 -8.67187,1.84375 -1.53125,0 -3.46875,-0.53125 -1.9375,-0.51563 -2.79688,-1.26563 z m 6,-25.40625 -1.64062,11.20313 -2,10.67187 q 0.0625,0.0469 0.14062,0.0781 1.84375,1.32812 4.375,1.32812 5.01563,0 7.375,-1.79687 2.35938,-1.79688 3.60938,-4.21875 1.25,-2.42188 1.25,-4.92188 0,-2.51562 -1.40625,-4.64062 -1.39063,-2.125 -3.76563,-3.57813 -2.35937,-1.46875 -7.9375,-4.125 z m 32.16406,24.28125 q -4.21875,2.03125 -6.01562,2.03125 -7.29688,0 -7.29688,-6.95312 0,-6.46875 4.07813,-10.46875 4.09375,-4 8.98437,-4 1.89063,0 3.90625,0.96875 2.03125,0.96875 2.03125,2.25 0,0.79687 -0.53125,1.25 -0.39062,0.76562 -1.10937,4.40625 -0.70313,3.64062 -0.70313,5.25 0,1.96875 1.01563,5.42187 l 0.0781,0.3125 q -0.4375,1.82813 -2.17187,1.82813 -0.29688,0 -1.09375,-0.79688 -0.79688,-0.79687 -1.17188,-1.5 z m 1.42188,-15.1875 q -1.21875,-0.73437 -1.95313,-0.73437 -3.25,0 -6.07812,2.95312 -2.82813,2.9375 -2.82813,7.3125 0,4.29688 3.71875,4.29688 2.90625,0 5.46875,-2.15625 0.3125,-8.1875 1.67188,-11.67188 z m 16.45312,-0.34375 q -0.23437,0.9375 -0.60937,2.1875 -1.64063,5.42188 -1.64063,8.26563 0,3.35937 1.5625,3.35937 1.54688,0 2.28125,-0.95312 1.01563,-1.32813 1.875,-1.32813 0.65625,0 1.25,0.48438 0.59375,0.48437 0.59375,1.26562 0,1.71875 -2.15625,2.95313 -2.15625,1.21875 -4.29687,1.21875 -5.03125,0 -5.03125,-6.28125 0,-3.4375 1.54687,-8.39063 0.40625,-1.3125 0.78125,-2.625 -4.25,-0.20312 -4.85937,-0.70312 -0.60938,-0.5 -0.60938,-1.5625 0,-0.85938 0.57813,-1.42188 0.59375,-0.57812 1.60937,-0.57812 l 4.03125,0.32812 q 0.125,-0.51562 0.1875,-1.17187 0.0312,-0.67188 0.125,-1.48438 l 0.34375,-2.42187 q 0.10938,-0.79688 0.71875,-1.34375 0.625,-0.54688 1.46875,-0.54688 1.64063,0 1.64063,2.625 0,0.76563 -0.23438,1.98438 l -0.39062,2.35937 q 2.60937,-0.32812 2.78125,-0.32812 2.03125,0 2.79687,0.40625 0.76563,0.39062 0.76563,1.46875 0,0.95312 -0.65625,1.54687 -0.65625,0.57813 -1.57813,0.57813 l -1.92187,-0.0781 q -1.45313,0 -2.95313,0.1875 z m 21.44141,15.53125 q -4.21875,2.03125 -6.01563,2.03125 -7.29687,0 -7.29687,-6.95312 0,-6.46875 4.07812,-10.46875 4.09375,-4 8.98438,-4 1.89062,0 3.90625,0.96875 2.03125,0.96875 2.03125,2.25 0,0.79687 -0.53125,1.25 -0.39063,0.76562 -1.10938,4.40625 -0.70312,3.64062 -0.70312,5.25 0,1.96875 1.01562,5.42187 l 0.0781,0.3125 q -0.4375,1.82813 -2.17188,1.82813 -0.29687,0 -1.09375,-0.79688 -0.79687,-0.79687 -1.17187,-1.5 z m 1.42187,-15.1875 q -1.21875,-0.73437 -1.95312,-0.73437 -3.25,0 -6.07813,2.95312 -2.82812,2.9375 -2.82812,7.3125 0,4.29688 3.71875,4.29688 2.90625,0 5.46875,-2.15625 0.3125,-8.1875 1.67187,-11.67188 z"
id="path4069"
inkscape:connector-curvature="0"
style="fill:#000000;fill-rule:nonzero" />
<path
d="m 299.38846,189.04987 h 70.36065 v -6.68504 l 8.40525,13.37008 -8.40525,13.37007 v -6.68502 h -70.36065 z"
id="path4071"
inkscape:connector-curvature="0"
style="fill:#7c7ce0;fill-rule:evenodd;stroke-width:0.79288208" />
<path
d="M 427.88773,353.66272 H 553.16596 V 468.46329 H 427.88773 Z"
id="path4075"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<g
transform="matrix(0.45555722,0,0,0.45555774,379.83689,353.66273)"
id="g4082">
<clipPath
id="p.3">
<path
d="M 0,0 H 275 V 252 H 0 Z"
id="path4077"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
<image
clip-path="url(#p.3)"
width="275"
height="252"
x="0"
y="0"
preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/magic.gif"
id="image4080"
style="fill:#000000" />
</g>
<path
d="m 660.00147,264.20093 21.66931,-21.66928 21.66932,21.66928 h -10.83466 v 85.7638 h 10.83466 l -21.66932,21.66928 -21.66931,-21.66928 h 10.83466 v -85.7638 z"
id="path4088"
inkscape:connector-curvature="0"
style="fill:#db4437;fill-rule:evenodd" />
<path
d="m 660.00147,264.20093 21.66931,-21.66928 21.66932,21.66928 h -10.83466 v 85.7638 h 10.83466 l -21.66932,21.66928 -21.66931,-21.66928 h 10.83466 v -85.7638 z"
id="path4090"
inkscape:connector-curvature="0"
style="fill-rule:evenodd;stroke:#666666;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
<path
d="m 554.53672,285.04017 c 42.66144,-12.09317 85.32288,12.0932 127.98438,0 42.66143,-12.09317 85.32287,12.0932 127.98431,0 l -1.70062,50.79135 c -42.66144,12.09317 -85.32288,-12.09317 -127.98432,0 -42.66143,12.09317 -85.32294,-12.09317 -127.98437,0 z"
id="path4092"
inkscape:connector-curvature="0"
style="fill:#db4437;fill-rule:evenodd" />
<path
d="m 587.72404,321.45273 q -0.84375,-0.5 -1.26562,-1.0625 -0.40625,-0.5625 -0.40625,-1.07812 0,-0.67188 0.67187,-1.07813 0.125,-0.0625 0.35938,-0.0625 0.3125,0 0.67187,0.10938 0.375,0.10937 0.57813,0.29687 0.82812,0.76563 1.98437,0.76563 0.5,0 1.46875,-0.21875 1.25,-0.39063 1.96875,-0.95313 0.71875,-0.57812 1.70313,-1.79687 0.60937,-0.79688 0.90625,-1.60938 0.3125,-0.82812 0.3125,-1.5 0,-0.73437 -0.34375,-1.15625 -0.64063,-0.85937 -1.4375,-1.28125 -0.78125,-0.4375 -1.82813,-0.4375 -0.9375,0 -1.82812,0.25 l -1.85938,0.39063 q -0.125,0.0312 -0.34375,0.0469 -0.20312,0.0156 -0.59375,0.95313 -0.375,0.92187 -1.29687,3.51562 -0.67188,1.875 -1.125,2.70313 -0.45313,0.8125 -0.79688,0.8125 -0.39062,0 -0.82812,-0.70313 -0.0937,-0.15625 -0.40625,-0.64062 -0.29688,-0.48438 -0.29688,-0.9375 0,-0.21875 0.15625,-0.53125 0.15625,-0.29688 0.26563,-0.57813 0.10937,-0.29687 0.25,-0.79687 0.40625,-1.25 2.40625,-6.20313 2,-4.96875 3.125,-7.5 0.79687,-1.82812 0.48437,-1.92187 0,-0.0937 -0.10937,-0.28125 -0.0937,-0.20313 -0.0625,-0.29688 0,-0.1875 1.14062,-0.5625 1.14063,-0.39062 1.54688,-0.32812 h 0.20312 q 0.40625,0 0.40625,-0.15625 0.0625,0 0.59375,-0.0625 0.53125,-0.0625 1.14063,0 2.26562,0.1875 3.57812,1.29687 1.3125,1.09375 1.28125,3.23438 -0.0625,0.84375 -0.92187,2.28125 -0.85938,1.4375 -1.89063,2.35937 -0.51562,0.45313 -1.17187,0.95313 -0.65625,0.5 -0.84375,0.625 l 0.79687,0.28125 q 1.5,0.57812 2.32813,1.42187 0.84375,0.82813 1.1875,2.26563 0.0781,0.3125 0.0781,0.57812 0,0.70313 -0.25,1.59375 -0.23437,0.89063 -0.53125,1.5625 -0.1875,0.20313 -0.3125,0.42188 -0.125,0.21875 -0.15625,0.28125 -0.0312,0.14062 -0.57812,0.98437 -0.54688,0.84375 -0.96875,1.35938 -2.10938,1.79687 -3.48438,2.29687 -1.34375,0.60938 -3.20312,0.60938 -1.65625,0 -2.45313,-0.51563 z m 2.29688,-13.26562 q 0.32812,-0.0781 1.28125,-0.28125 0.95312,-0.20313 1.375,-0.35938 0.0625,-0.0312 0.39062,-0.20312 0.34375,-0.1875 0.70313,-0.54688 2.01562,-1.875 2.8125,-3.39062 0.79687,-1.3125 0.79687,-1.84375 0,-0.39063 -0.34375,-0.75 -0.60937,-0.70313 -2.01562,-0.70313 -0.60938,0 -1.03125,0.14063 -0.45313,0.125 -0.65625,0.29687 -0.20313,0.17188 -0.20313,0.53125 0,0.25 -0.17187,0.5625 -0.15625,0.29688 -0.34375,0.48438 -0.0312,0.20312 -0.17188,0.39062 -0.14062,0.1875 -0.21875,0.25 0,0.0781 -0.4375,0.875 -0.42187,0.79688 -0.79687,1.78125 -0.96875,2.28125 -0.96875,2.76563 z m 11.90643,10.29687 q -0.0937,-0.1875 -0.54688,-0.5 -0.45312,-0.32812 -0.45312,-0.39062 0,-0.14063 -0.14063,-0.375 -0.14062,-0.23438 -0.26562,-0.4375 -0.26563,-0.25 0.34375,-1.65625 0.60937,-1.40625 1.64062,-3.32813 0.1875,-0.1875 0.48438,-0.67187 0.3125,-0.48438 0.40625,-0.54688 0,-0.125 0.0937,-0.23437 0.0937,-0.10938 0.23437,-0.10938 l 0.25,-0.51562 q 0.125,-0.29688 1.25,-1.375 1.125,-1.09375 1.76563,-1.46875 0.59375,-0.42188 1.3125,-0.42188 0.76562,0 1.29687,0.45313 l 0.875,0.73437 q 0.95313,0.79688 1.07813,0.92188 l 1.1875,1.0625 -0.54688,1.59375 q -0.28125,0.85937 -0.28125,1.64062 0,1.20313 0.67188,2.26563 0.15625,0.34375 0.42187,0.54687 0.28125,0.1875 0.73438,0.25 0.70312,0.125 0.70312,0.70313 0,0.70312 -0.26562,1.0625 -0.25,0.34375 -0.60938,0.4375 -0.5,0.0781 -0.70312,0.0781 -0.92188,0 -1.6875,-0.48437 -0.76563,-0.48438 -1.34375,-1.64063 -0.125,-0.1875 -0.34375,-0.6875 -0.20313,-0.51562 -0.23438,-0.71875 -0.21875,0.32813 -0.73437,0.84375 -3.29688,3.32813 -5.34375,3.32813 -0.70313,0 -1.25,-0.35938 z m 1.76562,-2.65625 q 0.5,0.0625 1.98438,-1.15625 0.15625,-0.125 0.34375,-0.29687 0.1875,-0.17188 0.42187,-0.375 0.89063,-0.73438 1.39063,-1.21875 0.5,-0.5 1,-1.20313 l 0.70312,-1.15625 q -0.3125,-0.70312 -0.57812,-0.9375 -0.25,-0.25 -0.5,-0.25 -0.64063,0 -1.60938,1.14063 -0.95312,1.14062 -2.29687,3.4375 -0.35938,0.57812 -0.57813,1.20312 -0.21875,0.625 -0.28125,0.8125 z m 14.94391,3.8125 q -0.78125,0 -1.59375,-0.32812 -0.8125,-0.3125 -1.375,-0.78125 -0.5625,-0.46875 -0.5625,-0.84375 0,-0.39063 0.35938,-0.48438 l 0.15625,-0.125 q 0.0625,-0.0312 0.25,-0.0312 0.14062,-0.0312 0.32812,0.0156 0.1875,0.0469 0.3125,0.0469 0.23438,0.0625 0.67188,0.125 l 0.84375,0.0937 q 0.64062,0 1.1875,-0.125 0.5625,-0.125 0.5625,-0.28125 0,-0.10937 -0.25,-0.32812 -0.25,-0.21875 -0.54688,-0.21875 0,-0.125 -0.98437,-0.85938 -0.54688,-0.42187 -1.04688,-0.82812 -0.5,-0.42188 -0.5,-0.625 l 0.0312,-0.0625 v -0.15625 q -0.21875,0 -0.42187,-0.60938 -0.1875,-0.60937 -0.1875,-1.1875 0,-1.07812 0.95312,-2.39062 0.96875,-1.3125 2.39063,-2.35938 1.4375,-1.0625 2.71875,-1.45312 0.46875,-0.21875 1.40625,-0.21875 0.60937,0 0.82812,0.125 l 0.25,-0.0625 q 0.39063,0 1,0.64062 0.60938,0.64063 0.76563,1.01563 0,1.0625 -0.0625,1.70312 -0.0625,0.64063 -0.32813,0.64063 -0.0312,0 -0.0937,0.0312 -0.0625,0.0312 -0.0625,0.15625 0,0.45312 -0.78125,1 -0.78125,0.53125 -1.42187,0.53125 l -0.125,-0.0312 q -0.20313,-0.0312 -0.57813,-0.375 -0.35937,-0.35938 -0.35937,-0.70313 0,-0.0937 0.375,-0.51562 0.39062,-0.42188 0.65625,-0.8125 0.28125,-0.40625 0.28125,-0.85938 0,-0.15625 -0.10938,-0.20312 -0.10937,-0.0469 -0.4375,-0.0469 -1.75,0.79688 -2.51562,1.5 -0.10938,0.20313 -0.1875,0.29688 -0.0781,0.0937 -0.20313,0.0937 0,0 0,-0.0156 0,-0.0156 -0.0312,-0.0156 -0.0625,0 -0.42187,0.375 -0.34375,0.35937 -0.65625,0.84375 -0.29688,0.46875 -0.35938,0.89062 l -0.0625,0.0937 v 0.15625 q -0.0312,0.0625 -0.0312,0.26563 0,0.28125 0.25,0.59375 0.25,0.29687 1.03125,0.84375 1.4375,1.09375 1.95313,1.625 0.53125,0.51562 0.67187,0.9375 0.28125,0.57812 0.28125,1.01562 0,1.09375 -1.14062,1.6875 -1.125,0.59375 -3.07813,0.59375 z m 13.21564,-13.79687 q -0.10937,0.0937 -0.20312,0.0937 -0.125,0 -0.40625,-0.1875 -0.32813,-0.0312 -0.85938,-0.54688 -0.51562,-0.51562 -0.51562,-0.67187 -0.0312,-0.39063 0.0625,-1.10938 0.0937,-0.71875 0.35937,-1.28125 0.28125,-0.57812 0.71875,-0.60937 0.23438,0.0312 0.60938,0.14062 0.39062,0.10938 0.875,0.29688 0.59375,1.21875 0.59375,2.07812 0,1.28125 -1.23438,1.79688 z m -2.95312,13.28125 q -0.76563,0 -1.32813,-0.79688 -0.5625,-0.79687 -0.6875,-1.98437 0.125,-1.1875 0.1875,-1.73438 0.35938,-1.5625 0.89063,-3.6875 0.53125,-2.14062 0.625,-2.35937 0.15625,-0.28125 0.5,-0.28125 0.35937,0 0.78125,0.26562 0.4375,0.26563 0.73437,0.75 0.25,0.35938 0.25,0.79688 0,0.67187 -0.64062,2.82812 -0.4375,1.59375 -0.57813,2.26563 -0.125,0.67187 -0.0937,1.4375 -0.125,1.4375 -0.25,1.98437 -0.125,0.54688 -0.39062,0.51563 z m 6.8189,-0.21875 q -0.95312,0 -1.25,-0.10938 -0.28125,-0.125 -0.70312,-0.5 -0.0937,-0.23437 -0.40625,-0.60937 -0.29688,-0.39063 -0.48438,-0.57813 -0.0937,-0.35937 -0.0937,-0.82812 0,-0.84375 0.23438,-1.95313 0.23437,-1.125 0.59375,-2.04687 0.125,-0.26563 0.20312,-0.48438 0.0781,-0.21875 0.10938,-0.28125 0,-0.23437 0.8125,-1.39062 0.82812,-1.17188 1.3125,-1.625 0.21875,-0.25 0.67187,-0.53125 0.46875,-0.29688 0.82813,-0.42188 0.0937,-0.0312 0.78125,-0.28125 0.6875,-0.26562 1.26562,-0.26562 0.51563,0 1,0.20312 0.5,0.20313 0.82813,0.65625 0.82812,0.82813 0.82812,2.17188 0,0.26562 -0.0625,0.57812 -0.17187,0.64063 -0.67187,1.04688 -0.48438,0.39062 -1.03125,0.39062 -0.54688,0 -0.92188,-0.40625 -0.125,-0.32812 -0.125,-0.42187 0,-0.0937 0.125,-0.54688 0.125,-0.57812 0.125,-0.73437 0,-0.40625 -0.28125,-0.40625 -0.39062,0 -0.625,0.1875 -0.21875,0.1875 -0.82812,0.82812 -0.40625,0.45313 -0.96875,1.29688 -0.5625,0.84375 -0.875,1.60937 -0.29688,0.875 -0.40625,1.25 -0.10938,0.35938 -0.10938,0.42188 0,0.125 -0.0312,0.23437 -0.0312,0.10938 -0.0312,0.25 0,0.21875 0.0937,0.34375 0.0937,0.125 0.34375,0.26563 0.0781,0.0312 0.23438,0.125 0.15625,0.0937 0.3125,0.0937 0.20312,0 0.54687,-0.15625 0.35938,-0.17188 0.45313,-0.20313 0.40625,-0.1875 1.17187,-0.82812 l 0.78125,-0.57813 q 0.21875,-0.15625 0.39063,-0.26562 0.1875,-0.10938 0.28125,-0.15625 0.0937,-0.0469 0.0937,0.0156 0,0.125 0.0937,0.125 0.0937,0 0.21875,-0.125 0.125,-0.20312 0.45312,-0.20312 0.25,0 0.51563,0.15625 0.28125,0.14062 0.28125,0.23437 l -0.0312,0.15625 q 0,0.15625 0.15625,0.32813 0,0.125 -0.0781,0.32812 -0.0781,0.20313 -0.20312,0.34375 -0.32813,0.46875 -0.73438,0.98438 -0.39062,0.51562 -0.54687,0.70312 h -0.0312 q -0.15625,0 -0.45313,0.29688 -0.46875,0.4375 -0.82812,0.4375 -0.125,0.0625 -0.29688,0.15625 -0.15625,0.0781 -0.15625,0.17187 -0.53125,0.54688 -2.875,0.54688 z m 10.59815,-0.32813 q -0.54688,0.125 -1.3125,-0.57812 -0.76563,-0.70313 -0.90625,-1.3125 -0.0625,-0.3125 -0.0625,-0.54688 0,-1.23437 0.79687,-3.01562 0.8125,-1.78125 2.17188,-3.34375 1.35937,-1.5625 2.98437,-2.26563 0.67188,-0.35937 0.85938,-0.35937 0.20312,0 0.875,0.35937 0.64062,0.15625 1.01562,0.34375 0.39063,0.1875 0.70313,0.60938 0.42187,0.82812 0.60937,2.04687 0.20313,0.67188 0.20313,0.95313 0,0.26562 -0.20313,0.45312 -0.0312,0.0312 -0.10937,0.125 -0.0781,0.0937 -0.0781,0.15625 0,0.0469 0.0625,0.0469 0,0.0312 -0.0312,0.14062 -0.0312,0.10938 -0.125,0.29688 -0.39062,1.21875 -0.39062,1.89062 0,0.21875 0.0625,0.54688 0.32812,0.82812 0.8125,0.82812 0.4375,0 1.01562,-0.57812 0.48438,-0.3125 0.96875,-0.3125 0.28125,0 0.28125,0.15625 0.0312,0.0312 0.125,0.0937 0.0937,0.0625 0.1875,0.0625 0.14063,0 0.14063,0.29687 0,0.25 -0.125,0.67188 -0.10938,0.40625 -0.26563,0.67187 -0.21875,0.21875 -0.35937,0.39063 -0.125,0.15625 -0.125,0.25 l -0.46875,0.4375 q -0.60938,0.35937 -1.45313,0.35937 -0.85937,0 -1.375,-0.35937 -0.5,-0.34375 -0.95312,-0.875 -0.45313,-0.53125 -0.57813,-0.9375 l -0.15625,-0.32813 -0.9375,0.60938 -0.79687,0.82812 q -0.25,0.32813 -0.4375,0.32813 h -0.32813 q 0,0.21875 -0.90625,0.54687 -0.89062,0.3125 -1.39062,0.3125 z m 0.67187,-2.6875 q 0.34375,0 1.4375,-0.8125 1.10938,-0.8125 1.5,-1.29687 0.125,-0.15625 0.32813,-0.45313 0.21875,-0.3125 0.375,-0.46875 0.51562,-0.60937 0.75,-0.96875 0.25,-0.375 0.25,-0.73437 0,-0.25 -0.23438,-0.70313 -0.125,-0.39062 -0.375,-0.39062 -0.14062,0 -0.73437,0.23437 -0.60938,0.34375 -1.4375,1.4375 -0.8125,1.07813 -1.42188,2.20313 -0.60937,1.125 -0.60937,1.64062 0,0.0937 0.0625,0.1875 0.0781,0.0937 0.10937,0.125 z m 11.06427,4.21875 q -0.48437,-0.15625 -0.71875,-0.57812 -0.23437,-0.40625 -0.23437,-1.20313 0,-1.3125 0.25,-2.5625 l 0.0312,-0.25 q 0.0312,-0.20312 0.0937,-0.625 0.0625,-0.4375 0.0937,-0.25 l 0.20312,-0.85937 q 0.21875,-0.89063 0.42188,-1.65625 0.21875,-0.78125 0.34375,-1.125 l 0.34375,-1.125 q 0.96875,-3.10938 1.79687,-4.1875 0.0312,-0.125 0.0937,-0.23438 0.0625,-0.125 0.0937,-0.25 0,-0.51562 0.20313,-0.79687 0.28125,-0.48438 0.46875,-0.85938 l 0.14062,-0.35937 0.89063,-1.82813 q 0.0937,-0.3125 0.39062,-0.6875 0.3125,-0.39062 0.625,-0.53125 l -0.0312,-0.0312 q 0.17188,-0.25 0.59375,-0.4375 0.4375,-0.20312 0.75,-0.20312 0.42188,0 0.67188,0.26562 0.26562,0.25 0.26562,0.70313 v 0.3125 q 0,0.29687 -0.0469,0.45312 -0.0469,0.15625 -0.21875,0.39063 h 0.0312 q 0.0469,0.0312 -0.0469,0.20312 -0.0781,0.17188 -0.14062,0.32813 -0.21875,0.57812 -0.3125,0.64062 -0.23438,0.29688 -1.48438,2.71875 -1.23437,2.40625 -1.59375,3.32813 l -0.125,0.48437 -0.0625,0.20313 q -0.35937,0.76562 -0.39062,0.82812 l -0.51563,1.98438 -0.125,0.67187 -0.28125,0.73438 -0.1875,0.60937 q -0.0781,0.15625 -0.125,0.42188 -0.0469,0.25 -0.10937,0.3125 l -0.0312,0.15625 q -0.0312,0.125 -0.21875,0.73437 -0.29688,0.90625 -0.42188,1.5 -0.125,0.57813 -0.15625,1.25 l -0.0312,0.14063 q -0.0937,0.70312 -0.40625,1.04687 -0.29687,0.35938 -0.78125,0.21875 z m 6.82099,-0.59375 q -0.0937,0.0312 -0.26563,0.0312 -0.67187,0 -1.125,-0.76562 -0.4375,-0.78125 -0.4375,-1.96875 0,-0.625 0.125,-1.23438 0.35938,-0.9375 0.35938,-1.34375 0.0937,-0.60937 0.65625,-2.1875 0.5625,-1.59375 0.71875,-1.92187 0.125,0 0.21875,-0.20313 0.0937,-0.20312 0.0937,-0.46875 0,-0.28125 0.10937,-0.57812 0.125,-0.3125 0.25,-0.4375 0,-0.0937 0.0781,-0.34375 0.0781,-0.26563 0.0781,-0.35938 0,-0.0625 0.0781,-0.23437 0.0781,-0.1875 0.0781,-0.28125 0.51563,-0.98438 1.79688,-4.45313 1.28125,-3.48437 1.28125,-3.85937 0,-0.26563 0.1875,-0.48438 0.20312,-0.21875 0.45312,-0.21875 0.0937,-0.10937 0.40625,-0.15625 0.32813,-0.0469 0.51563,-0.0469 l 0.51562,0.51562 q 0.35938,0.32813 0.54688,0.57813 0.1875,0.25 0.1875,0.60937 0,0.57813 -0.20313,1.14063 -0.20312,0.54687 -1.20312,2.625 -0.0625,0.17187 -0.51563,1.15625 -0.3125,0.64062 -0.3125,0.73437 0,0.0937 -0.14062,0.40625 -0.125,0.29688 -0.21875,0.625 -0.25,0.51563 -0.8125,1.8125 -0.5625,1.29688 -0.5625,1.42188 0,0.34375 -0.15625,0.5 -0.0937,0 -0.125,0.15625 -0.0312,0.14062 -0.0312,0.20312 0,0.3125 -0.51563,1.70313 l -0.28125,0.92187 q -0.60937,2.07813 -0.85937,3.04688 -0.23438,0.95312 -0.23438,1.625 0,0.70312 -0.23437,1.15625 -0.21875,0.4375 -0.5,0.57812 z m 13.97729,-12.23437 q 0.29688,0.57812 0.29688,1.3125 0,0.9375 -0.42188,1.53125 -0.3125,0.45312 -0.5625,1.35937 -0.23437,0.89063 -0.45312,2.39063 -0.29688,1.21875 -0.71875,3.75 -0.5,1.34375 -0.625,2.01562 -0.0469,0.1875 -0.3125,0.79688 -0.26563,0.60937 -0.4375,0.95312 -1.4375,2.92188 -3.40625,4.39063 -1.96875,1.46875 -4.01563,1.59375 -0.51562,0.0625 -0.60937,0.0625 -0.28125,0 -0.53125,-0.0937 -0.23438,-0.0937 -0.51563,-0.25 -0.64062,-0.45313 -0.73437,-0.64063 -0.17188,-0.32812 -0.17188,-0.64062 0,-0.48438 0.45313,-0.65625 0.45312,-0.17188 1.03125,-0.17188 0.21875,0.125 0.76562,0.125 0.54688,0 0.64063,-0.125 0.0625,-0.0625 0.375,-0.21875 0.32812,-0.14062 0.73437,-0.23437 1.0625,-0.51563 1.625,-1.03125 0.5625,-0.5 1.23438,-1.625 0.79687,-1.21875 1.04687,-2.20313 0.42188,-1.0625 1,-3.14062 l -0.0312,0.0312 q -0.17187,0.1875 -0.8125,0.5625 -0.625,0.375 -0.76562,0.375 -0.0312,0 -0.0937,0.0312 -0.0625,0.0312 -0.0625,0.125 0,0.125 -0.82813,0.40625 -0.82812,0.26562 -1.0625,0.26562 -0.3125,0 -0.6875,-0.34375 -0.35937,-0.35937 -0.53125,-0.65625 -0.125,-0.28125 -0.46875,-0.57812 -0.39062,-0.375 -0.57812,-0.67188 -0.1875,-0.3125 -0.1875,-0.73437 0,-0.21875 0.0312,-0.34375 0.375,-1.4375 1.40625,-4.42188 1.04687,-3 1.53125,-3.73437 0.125,-0.20313 0.46875,-0.32813 0.35937,-0.125 0.71875,-0.125 0.64062,0 0.85937,0.375 0.39063,0.54688 0.39063,0.9375 0,0.4375 -0.45313,1.3125 -0.125,0.34375 -0.32812,0.76563 -0.1875,0.42187 -0.1875,0.51562 -0.0312,0.125 -0.1875,0.78125 -0.15625,0.65625 -0.28125,0.65625 -0.20313,0.1875 -0.4375,1.32813 -0.23438,1.14062 -0.23438,1.8125 0,0.375 0.0937,0.375 0.85938,0 1.65625,-0.45313 0.79688,-0.46875 1.67188,-1.5 0.125,-0.79687 0.35937,-1.71875 0.25,-0.9375 0.34375,-1.25 0.57813,-1.98437 0.76563,-3.20312 0.1875,-0.45313 0.28125,-0.53125 0.0937,-0.0781 0.42187,-0.0781 0.39063,0 0.84375,0.23438 0.46875,0.21875 0.6875,0.53125 z m 11.99976,13.79687 q -1.25,-0.0312 -1.78125,-0.59375 -0.51563,-0.5625 -0.51563,-1.57812 0,-0.26563 0.0625,-0.84375 0.125,-0.4375 0.1875,-0.89063 0.15625,-0.70312 0.32813,-0.82812 0.0312,-0.0312 0.0625,-0.125 0.0312,-0.10938 -0.0625,-0.17188 0,-0.40625 0.28125,-1.14062 0.0625,-0.23438 0.15625,-0.45313 0.0937,-0.21875 0.14062,-0.28125 -0.0781,-0.0781 -0.0781,-0.20312 0,-0.15625 0.17187,-0.3125 0.1875,-0.15625 0.26563,-0.28125 0.0781,-0.14063 -0.0156,-0.26563 -0.0625,-0.15625 0.60938,-1.4375 0.42187,-0.82812 0.45312,-0.9375 -0.0312,-0.0937 0.45313,-1.17187 0.5,-1.09375 0.79687,-1.64063 0.28125,-0.15625 -0.23437,-0.21875 -0.51563,-0.0625 -1.25,-0.0312 -1.04688,-0.0312 -1.625,-0.10937 -0.57813,-0.0937 -0.9375,-0.28125 -0.21875,-0.21875 -0.21875,-0.57813 0,-0.40625 0.29687,-0.75 0.3125,-0.35937 0.76563,-0.39062 0.82812,-0.0312 2.5625,-0.23438 l 1.625,-0.125 0.0937,-0.54687 q 0.15625,-0.28125 0.29687,-0.48438 0.15625,-0.21875 0.20313,-0.375 0.0469,-0.15625 0.0781,-0.48437 0.0937,-0.125 0.32813,-0.625 0.25,-0.51563 0.3125,-1 0.35937,-0.60938 0.65625,-1.17188 0.29687,-0.5625 0.375,-0.65625 0.5625,-0.85937 1.26562,-0.85937 0.39063,0 0.8125,0.25 0.40625,0.29687 0.625,0.51562 0.23438,0.21875 0.23438,0.60938 0,0.28125 -0.29688,0.82812 -0.125,0.25 -0.375,1.03125 -0.21875,0.28125 -0.48437,0.78125 -0.25,0.5 -0.34375,0.9375 -0.0625,0.15625 -0.20313,0.4375 -0.125,0.26563 -0.17187,0.3125 -0.0469,0.0469 -0.14063,0.0469 0.0312,0.20312 0.125,0.26562 0.0937,0.0625 0.32813,0.0625 0.15625,0 0.39062,-0.0312 0.25,-0.0312 0.59375,-0.0312 0.32813,-0.0312 0.5625,-0.0937 0.23438,-0.0625 0.46875,-0.0625 0.125,0 0.375,0.0937 0.15625,0.1875 0.15625,0.73438 0,0.40625 -0.125,0.78125 -0.125,0.375 -0.3125,0.46875 -0.15625,0.0312 -0.70312,0.1875 -0.54688,0.15625 -1.1875,0.1875 l -1.70313,0.15625 -0.53125,1.09375 q -0.23437,0.39062 -0.39062,0.75 -0.15625,0.34375 -0.21875,0.40625 0,0.15625 -0.26563,0.64062 -0.25,0.48438 -0.375,0.76563 -0.17187,0.45312 -0.39062,0.89062 -0.21875,0.42188 -0.28125,0.54688 -0.0937,0 -0.125,0.0469 -0.0156,0.0469 -0.0156,0.14063 l 0.0781,0.26562 q 0,0.15625 -0.17188,0.3125 -0.125,0.0625 -0.23437,0.23438 -0.10938,0.15625 -0.14063,0.21875 0.0625,0.0625 0.0625,0.15625 0,0.23437 -0.21875,0.57812 0,0.54688 -0.45312,1.625 0.0312,0.57813 0.0625,0.75 0.0312,0.15625 0.15625,0.125 h 0.15625 l 0.70312,-0.0312 q 0.23438,-0.0937 0.10938,0.48438 -0.10938,0.5625 -0.54688,1.75 -0.48437,0.73437 -1.01562,1.29687 -0.51563,0.5625 -0.6875,0.5625 z m 8.84204,-3.14062 q -0.57813,-0.0937 -1.0625,-0.64063 -0.48438,-0.54687 -0.48438,-1.25 0,-0.76562 0.96875,-3.9375 0.15625,-0.82812 0.70313,-2.14062 0.1875,-0.54688 0.21875,-0.64063 0.57812,-1.92187 1.57812,-4.70312 l 0.28125,-0.76563 q 1.125,-3.04687 1.34375,-3.71875 l 0.51563,-1.40625 q 0,-0.25 0.35937,-0.48437 0.375,-0.25 0.625,-0.25 0.51563,0 0.98438,0.67187 0.48437,0.67188 0.51562,1.40625 0,0.39063 -0.15625,0.70313 -0.73437,1.60937 -1.82812,4.96875 -0.46875,1.40625 -0.95313,2.67187 -0.48437,1.26563 -0.76562,1.8125 l -0.35938,0.70313 -0.4375,1.6875 q 0,0.0937 -0.0312,0.17187 -0.0312,0.0625 -0.0312,0.0937 0,0.0937 0.0312,0.0937 l 0.53125,-0.54687 q 0.9375,-0.92188 2.21875,-1.89063 1.04688,-0.73437 1.07813,-0.82812 0.0312,-0.0937 0.26562,-0.29688 l 0.3125,-0.25 q 1.03125,-0.73437 2.21875,-0.73437 0.34375,0 0.82813,0.35937 0.48437,0.34375 0.8125,0.96875 0.34375,0.625 0.34375,1.32813 v 0.25 0.23437 q 0,0.79688 -0.3125,2.09375 -0.29688,1.29688 -0.32813,1.48438 0.15625,0 0.46875,-0.0625 0.48438,0 0.6875,0.0937 0.21875,0.0937 0.375,0.26562 0.15625,0.15625 0.28125,0.1875 0.0625,0.0625 0.0625,0.1875 0,0.73438 -0.79687,1.28125 -0.79688,0.54688 -1.53125,0.60938 -0.89063,0 -1.4375,-0.67188 -0.54688,-0.67187 -0.54688,-1.64062 0,-0.375 0.0312,-0.75 0.0312,-0.39063 0.0625,-0.6875 0.0937,-0.64063 0.0937,-1.01563 0,-0.42187 -0.0781,-0.5625 -0.0781,-0.14062 -0.26563,-0.14062 -0.64062,0 -2.23437,1.17187 -1.57813,1.15625 -3.20313,3.01563 -1.03125,1.5 -1.95312,1.5 z m 17.8548,1.21875 q -1.82813,0 -2.95313,-1.01563 -1.10937,-1.03125 -1.10937,-2.92187 0,-0.73438 0.0937,-1.625 0.0625,-0.78125 0.625,-2.125 0.5625,-1.34375 1.53125,-2.60938 0.96875,-1.28125 2.28125,-1.89062 0.60937,-0.29688 1.03125,-0.40625 0.42187,-0.10938 0.95312,-0.10938 1.3125,0 2.125,0.48438 0.82813,0.46875 1.15625,1.07812 0.34375,0.60938 0.34375,1.0625 v 0.15625 0.0937 q 0,0.39063 -0.45312,1.42188 -0.45313,1.01562 -1.42188,1.92187 -0.96875,0.89063 -2.51562,0.98438 -0.76563,0 -1.48438,-0.14063 -0.71875,-0.15625 -0.90625,-0.15625 -0.48437,0 -0.60937,0.35938 -0.125,0.34375 -0.125,1.1875 0.0937,0.89062 0.48437,1.25 0.40625,0.34375 1.17188,0.34375 0.39062,0 0.89062,-0.21875 0.51563,-0.26563 1.21875,-0.8125 0.39063,-0.3125 0.60938,-0.4375 0.21875,-0.125 0.54687,-0.125 0.28125,0 0.67188,0.39062 0.39062,0.375 0.42187,0.71875 -0.0781,0.39063 -0.71875,1.15625 -0.625,0.76563 -1.65625,1.375 -1.03125,0.60938 -2.20312,0.60938 z m 1.10937,-7.71875 q 0.60938,0 0.84375,-0.125 0.3125,-0.15625 0.96875,-0.8125 0.65625,-0.65625 0.6875,-0.90625 0,-0.32813 -0.28125,-0.53125 -0.26562,-0.20313 -0.73437,-0.28125 h -0.35938 q -0.42187,0 -1.20312,0.70312 -0.78125,0.6875 -1.35938,1.57813 0.0312,0.15625 0.48438,0.26562 0.45312,0.10938 0.95312,0.10938 z m 16.38513,7.90625 q -0.98437,0 -1.85937,-0.4375 -0.85938,-0.45313 -1.375,-1.14063 -0.5,-0.6875 -0.5,-1.35937 0,-0.54688 0.4375,-0.54688 0.29687,0.0625 0.67187,0.35938 0.48438,0.28125 0.8125,0.4375 0.34375,0.15625 0.70313,0.15625 0.53125,0 0.9375,-0.17188 0.40625,-0.17187 0.46875,-0.53125 0,-0.125 -0.14063,-0.54687 -0.125,-0.4375 -0.3125,-0.79688 -0.70312,-1.04687 -0.95312,-2.04687 -0.23438,-1.01563 -0.29688,-2.35938 0,-0.73437 1.0625,-1.75 1.07813,-1.03125 2.5,-1.82812 1.4375,-0.8125 2.32813,-0.90625 1.76562,0 2.10937,0.29687 0.45313,0.34375 0.82813,1.14063 0.39062,0.79687 0.39062,1.34375 0,0.23437 -0.21875,0.57812 -0.20312,0.32813 -0.26562,0.42188 l -0.28125,0.4375 -0.21875,0.48437 q -0.0781,0.15625 -0.21875,0.48438 -0.14063,0.3125 -0.25,0.39062 -0.10938,0.0625 -0.34375,0.0156 -0.53125,0 -0.98438,-0.46875 -0.45312,-0.48438 -0.45312,-1.0625 0,-0.40625 0.28125,-0.82813 0.29687,-0.42187 0.29687,-0.48437 0,-0.125 -0.28125,-0.125 -0.42187,0 -1.29687,0.42187 -0.85938,0.40625 -1.53125,1.09375 -0.67188,0.6875 -0.67188,1.53125 0,0.4375 0.17188,1.03125 0.1875,0.59375 0.40625,0.98438 0.26562,0.25 0.75,1.51562 0.5,1.26563 0.5,2 0,1.125 -0.84375,1.70313 -0.84375,0.5625 -2.35938,0.5625 z m 11.04377,-0.64063 q -0.3125,0.0625 -0.57813,0.0625 -0.40625,0 -0.76562,-0.0781 -0.34375,-0.0781 -0.34375,-0.20313 0,-0.0937 -0.0469,-0.125 -0.0312,-0.0312 -0.125,-0.0312 -0.0937,-0.0937 -0.4375,-0.32812 -0.32812,-0.25 -0.54687,-0.34375 -0.39063,-0.54688 -0.39063,-1.1875 0,-0.60938 0.39063,-1.95313 0.0312,-1.25 0.90625,-2.79687 0.875,-1.54688 2.125,-2.875 1.25,-1.32813 2.25,-1.90625 0.46875,-0.48438 1.34375,-0.48438 0.67187,0 1.1875,0.28125 0.40625,0.26563 1.03125,1.09375 0.625,0.82813 0.9375,1.5 0.0781,0.23438 0.1875,0.625 0.10937,0.375 0.10937,0.92188 l -0.0625,0.89062 q -0.39062,1.3125 -0.39062,2.35938 -0.0625,0.45312 0.0625,0.73437 0.14062,0.26563 0.35937,0.26563 0.15625,0 0.34375,-0.0937 0.20313,-0.0937 0.45313,-0.0937 0.42187,0 0.70312,0.25 0.29688,0.25 0.29688,0.64062 0,0.3125 -0.17188,0.60938 -0.28125,0.64062 -1.0625,1.15625 -0.78125,0.5 -1.45312,0.5 -0.54688,0 -1.04688,-0.5625 -0.5,-0.57813 -0.78125,-1.51563 l -0.1875,-0.73437 -1.15625,1.0625 -0.73437,0.73437 q -0.51563,0.51563 -0.57813,0.60938 -0.15625,0.25 -0.5625,0.48437 -0.40625,0.21875 -1.26562,0.53125 z m 0.85937,-3.0625 q 0.57813,-0.32812 0.96875,-0.79687 0.40625,-0.48438 1.01563,-1.34375 0.14062,-0.20313 0.26562,-0.40625 0.125,-0.20313 0.3125,-0.4375 0.54688,-0.73438 0.78125,-1.17188 0.25,-0.45312 0.375,-0.96875 -0.125,-0.42187 -0.375,-0.78125 -0.23437,-0.375 -0.42187,-0.40625 -0.90625,0 -2.4375,2.10938 -0.15625,0.23437 -0.5,0.8125 -0.32813,0.5625 -0.42188,0.76562 -0.17187,0.21875 -0.4375,1.07813 -0.26562,0.84375 -0.26562,1.09375 0,0.70312 0.40625,0.70312 0.32812,0 0.73437,-0.25 z m 25.32318,0.92188 q 0.21875,0.32812 0.21875,0.60937 0,0.32813 -0.23437,0.57813 -0.21875,0.25 -0.375,0.25 -0.76563,0 -1.71875,-0.35938 -0.9375,-0.375 -0.9375,-1.07812 v -1.98438 q 0,-1.28125 -0.21875,-1.46875 -0.48438,-0.25 -1.28125,0.71875 -0.79688,0.96875 -1.95313,2.89063 -0.45312,0.70312 -0.78125,1.01562 -0.3125,0.29688 -0.5625,0.29688 -0.21875,0 -0.39062,-0.125 -0.98438,-0.28125 -0.98438,-1.125 0,-0.0937 0.0625,-0.40625 0.28125,-1 0.60938,-2.20313 0.34375,-1.21875 0.54687,-2.09375 -0.0937,-0.1875 -0.32812,-0.1875 -0.3125,0 -0.73438,0.32813 -0.125,0.0937 -0.46875,0.29687 -0.32812,0.20313 -0.67187,0.54688 -0.32813,0.32812 -0.6875,0.90625 -0.125,0.23437 -0.4375,0.67187 -1.60938,2.4375 -1.95313,2.75 -0.17187,0.45313 -0.3125,0.64063 -0.14062,0.17187 -0.32812,0.17187 -0.125,0 -0.32813,-0.10937 -0.60937,-0.28125 -0.98437,-0.8125 -0.35938,-0.53125 -0.35938,-1.26563 0,-0.46875 0.28125,-1.90625 0.29688,-1.45312 0.65625,-2.70312 0.25,-0.5 0.39063,-1.0625 0.14062,-0.5625 0.14062,-0.78125 0.10938,-0.60938 0.39063,-1.64063 0.28125,-1.04687 0.42187,-1.15625 0.1875,-0.1875 0.70313,-0.1875 0.34375,0 0.65625,0.14063 0.32812,0.125 0.45312,0.375 0.42188,0.48437 0.42188,1.3125 0,0.76562 -0.3125,1.625 -0.32813,0.73437 -0.8125,2.15625 0,0 0.0625,-0.0781 0.78125,-1.07812 1.98437,-2.42187 0.57813,-0.64063 1.29688,-0.98438 0.71875,-0.35937 1.17187,-0.35937 0.76563,0.125 1.29688,0.75 0.53125,0.625 0.65625,1.70312 0,0.39063 -0.0625,0.95313 -0.0625,0.5625 -0.0937,0.71875 0.125,-0.15625 0.65625,-0.78125 0.53125,-0.625 1.29687,-1.29688 0.70313,-0.54687 1.3125,-0.54687 0.5,0 0.89063,0.42187 0.70312,0.40625 1.09375,1.34375 0.40625,0.92188 0.40625,2.10938 0,0.40625 -0.0312,0.64062 -0.0312,0.21875 -0.0312,0.64063 0,0.4375 0.0937,0.84375 0.10937,0.40625 0.20312,0.71875 z m 4.44238,2.3125 q -1.0625,-0.42188 -1.5625,-1.29688 -0.48437,-0.89062 -0.48437,-2.29687 0,-0.3125 0.0625,-1.15625 0.0312,-0.25 0.0312,-0.59375 0,-0.54688 -0.125,-0.8125 -0.125,-0.28125 -0.45313,-0.57813 -0.0937,-0.0937 -0.45312,-0.40625 -0.34375,-0.32812 -0.3125,-0.48437 v -0.0937 q 0,-0.21875 0.10937,-0.48438 0.10938,-0.28125 0.21875,-0.34375 0.0937,-0.0312 0.375,-0.1875 0.29688,-0.17187 0.45313,-0.10937 0.125,0 0.39062,0.17187 0.28125,0.15625 0.28125,0.28125 0.1875,0.39063 0.98438,-1.25 0.45312,-0.67187 1.21875,-1.32812 0.76562,-0.65625 1.40625,-1 0.23437,-0.10938 1.01562,-0.28125 0.78125,-0.17188 1.23438,-0.17188 0.5,0 1.1875,0.42188 0.6875,0.40625 1.14062,0.89062 0.67188,0.48438 0.67188,1.82813 0,0.79687 -0.25,1.375 -0.125,0.46875 -1.26563,1.45312 -1.14062,0.96875 -1.84375,1.26563 -0.70312,0.15625 -1.5625,0.15625 -0.875,0 -1.1875,-0.15625 -0.21875,-0.125 -0.375,0.10937 -0.14062,0.23438 -0.23437,1.07813 l -0.0312,0.3125 q 0,0.35937 0.15625,0.67187 0.23437,0.39063 0.42187,0.54688 0.1875,0.15625 0.57813,0.15625 l 0.28125,-0.0312 q 0.35937,-0.0312 0.75,-0.34375 0.40625,-0.32813 0.95312,-0.875 0.57813,-0.54688 0.67188,-0.64063 0.15625,-0.3125 0.85937,-0.3125 0.28125,0 0.73438,0.0937 0.32812,0.25 0.32812,0.64063 0,0.54687 -0.51562,1.46875 -0.15625,0.32812 -0.98438,1.09375 -0.8125,0.76562 -1.29687,1.04687 -0.85938,0.48438 -1.90625,0.48438 -0.8125,0 -1.67188,-0.3125 z m 2.375,-7.20313 q 0.76563,-0.20312 1.1875,-0.46875 0.4375,-0.26562 1.01563,-1.07812 0.25,-0.76563 0.0937,-0.92188 -0.1875,-0.1875 -0.45313,-0.1875 -0.3125,0 -0.76562,0.34375 -0.3125,0.0937 -0.875,0.59375 -0.5625,0.5 -0.95313,0.96875 -0.375,0.45313 -0.21875,0.54688 -0.0625,0.10937 0.3125,0.15625 0.39063,0.0469 0.65625,0.0469 z"
id="path4094"
inkscape:connector-curvature="0"
style="fill:#000000;fill-rule:nonzero" />
<path
d="M 427.809,138.3307 H 553.08723 V 253.13125 H 427.809 Z"
id="path4100"
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:0;fill-rule:evenodd" />
<path
d="M 379.751,138.43701 H 505.04235 V 253.24016 H 379.751 Z"
id="path4109"
inkscape:connector-curvature="0"
style="fill:#00fdc8;fill-rule:evenodd" />
<path
d="M 379.751,138.43701 H 505.04235 V 253.24016 H 379.751 Z"
id="path4111"
inkscape:connector-curvature="0"
style="fill-rule:evenodd;stroke:#666666;stroke-width:1;stroke-linecap:butt;stroke-linejoin:round" />
<path
d="m 299.38583,404.37924 h 70.36065 v -6.68504 l 8.40525,13.37008 -8.40525,13.37007 v -6.68502 h -70.36065 z"
id="path4071-3"
inkscape:connector-curvature="0"
style="fill:#7c7ce0;fill-rule:evenodd;stroke-width:0.79288208" />
<path
d="m 506.71874,190.09462 h 70.36065 v -6.68504 l 8.40525,13.37008 -8.40525,13.37007 v -6.68502 h -70.36065 z"
id="path4071-6"
inkscape:connector-curvature="0"
style="fill:#7c7ce0;fill-rule:evenodd;stroke-width:0.79288208" />
<path
d="m 506.71875,404.41666 h 70.36065 v -6.68504 l 8.40525,13.37008 -8.40525,13.37007 v -6.68502 h -70.36065 z"
id="path4071-6-7"
inkscape:connector-curvature="0"
style="fill:#7c7ce0;fill-rule:evenodd;stroke-width:0.79288208" />
<flowRoot
xml:space="preserve"
id="flowRoot4272"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:26.66666603px;line-height:1.25;font-family:'Comic Sans MS';-inkscape-font-specification:'Comic Sans MS';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none"><flowRegion
id="flowRegion4274"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:26.66666603px;font-family:'Comic Sans MS';-inkscape-font-specification:'Comic Sans MS';text-align:center;text-anchor:middle"><rect
id="rect4276"
width="125.97456"
height="90.782555"
x="379.06781"
y="162.45763"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:26.66666603px;font-family:'Comic Sans MS';-inkscape-font-specification:'Comic Sans MS';text-align:center;text-anchor:middle" /></flowRegion><flowPara
id="flowPara4278">Some process</flowPara></flowRoot> <g
transform="matrix(0.45555722,0,0,0.45555774,379.751,138.43701)"
id="g4082-1"
inkscape:transform-center-x="-34.322034"
inkscape:transform-center-y="102.9661">
<clipPath
id="clipPath4292">
<path
d="M 0,0 H 275 V 252 H 0 Z"
id="path4290"
inkscape:connector-curvature="0"
style="clip-rule:evenodd" />
</clipPath>
<image
clip-path="url(#p.3-4)"
width="275"
height="252"
x="0"
y="0"
preserveAspectRatio="none"
xlink:href="https://desfontain.es/privacy/images/magic.gif"
id="image4080-3"
style="fill:#000000" />
</g>
<rect
style="fill:#ffffff"
id="rect148"
width="28.983051"
height="45.762711"
x="212.03391"
y="388.22034" />
</g>
</svg>
</center></p>
<p>By "basically the same", I don't mean "it looks a bit similar". Instead,
remember that the magic you added to the process was <em>randomized</em>. You don't
always get the same output if you run the new process several times. So what
does "basically the same" means in this context? It means that you can get the
<em>exact same</em> output from both databases with similar likelihood.</p>
<p>What does this have to do with privacy? Well, suppose you're a creepy person
trying to figure out whether your target is in the original data. By looking at
the output, you can't be 100% certain of anything. Sure, it could have come from
a database with your target in it. But it could also have come from the exact
same database, without your target. Both options have a similar probability, so
there's not much you can say.</p>
<p>You might have noticed that this definition doesn't say anything about what the
output data looks like. Differential privacy is not a property of the output
data. It's very different from, say, <a href="k-anonymity.html"><span class="math">\(k\)</span>-anonymity</a>, one of the first
data privacy definitions. You can't look at the output data and determine
whether it satisfies differential privacy. Instead, differential privacy is a
property of the <em>process</em>: you have to know how the data was generated to
determine whether it's differentially private.</p>
<p>That's about it for the high-level intuition. It's a little abstract, but not
very complicated. So, why all the hype? What makes it so awesome compared to
older, more straightforward definitions?</p>
<h1 id="what-makes-differential-privacy-special">What makes differential privacy special</h1>
<p>Privacy experts, especially in academia, are enthusiastic about differential
privacy. It was first proposed by Cynthia Dwork, Frank McSherry, Kobbi Nissim
and Adam Smith in 2006<sup id="fnref:papers"><a class="footnote-ref" href="#fn:papers">1</a></sup>. Very soon, almost all researchers working on
anonymization started building differentially private algorithms. Tech companies
and governments are adopting it fast. So, why all the hype? I can count three
main reasons.</p>
<h2 id="you-no-longer-need-attack-modeling">You no longer need attack modeling</h2>
<p>All definitions that came before needed some assumptions about the attacker. To
choose the right notion, you needed to figure out the attacker's capabilities
and goals. How much prior knowledge do they have? What auxiliary data are they
allowed to use? What kind of information do they want to learn?</p>
<p>Doing in practice was difficult and very error-prone. Answering these questions
is very tricky: in particular, you might not know exactly what the attacker
wants or is capable of. Worse, there might be <em>unknown unknowns</em>: attack vectors
that you didn't anticipate at all. For that reason, you couldn't make very broad
statements with these old-school definitions. You had to make some assumptions,
which you couldn't be 100% sure of.</p>
<p>By contrast, when you use differential privacy, you get two awesome guarantees.</p>
<ol>
<li>You protect <em>any kind of information about an individual</em>. It doesn't matter
what the attacker wants to do. Reidentify their target, know if they're in
the dataset, deduce some sensitive attribute… All those things are protected.
Thus, you don't have to think about the goals of your attacker.</li>
<li>It works <em>no matter what the attacker knows about your data</em>. They might
already know some people in the database. They might even add some fake users
to your system. With differential privacy, it doesn't matter. The users that
the attacker doesn't know are still protected.</li>
</ol>
<h2 id="you-can-quantify-the-privacy-loss">You can quantify the privacy loss <a name="quantify"></a></h2>
<p>Differential privacy, like older notions, comes with a numeric parameter that
you can tweak. There is a big difference, though, in how meaningful that
parameter is. Take <a href="k-anonymity.html"><span class="math">\(k\)</span>-anonymity</a>, for example. It tells you that each
record in the output dataset "looks like" at least <span class="math">\(k-1\)</span> other records. But does
the value of <span class="math">\(k\)</span> tell us about the level of protection?</p>
<p>The answer is… not much. There is no clear link between the value of <span class="math">\(k\)</span> and how
private the dataset is. So <a href="k-anonymity.html#how-to-choose-k">choosing <span class="math">\(k\)</span></a> is very handwavy, and can't be
justified in a formal way. The problem is <a href="l-diversity.html#the-bad-news-policy">even worse</a> with other old-school
definitions.</p>
<p>Differential privacy is much better. When you use it, you can quantify the
<em>greatest possible information gain</em> by the attacker. The corresponding
parameter, named <span class="math">\(\varepsilon\)</span>, allows you to make formal statements. Suppose
<span class="math">\(\varepsilon=1.1\)</span>. Then, you can say: "an attacker who thinks their target is in
the dataset with probability 50% can increase their level of certainty to at
most 75%." Choosing the exact value of <span class="math">\(\varepsilon\)</span> isn't easy, but at least,
it can be interpreted in a formal way.</p>
<p>And do you remember the previous point about attack modeling? It means you can
change this statement in many ways. You can replace "their target is is the
dataset" by anything about one individual. And you can add "no matter what the
attacker knows" if you want to be extra-precise. Altogether, that makes
differential privacy much stronger than all definitions that came before.</p>
<h2 id="you-can-compose-multiple-mechanisms">You can compose multiple mechanisms <a name="composition"></a></h2>
<p>Suppose you have some data. You want to share it with Alex and with Brinn, in
some anonymized fashion. You trust Alex and Brinn equally, so you use the same
definition of privacy for both of them. They are not interested in the same
aspects of the data, so you give them two different versions of your data. Both
versions are "anonymous", for the definition you've chosen.</p>
<p>What happens if Alex and Brinn decide to conspire, and compare the data you gave
them? Will the union of the two anonymized versions still be anonymous? It turns
out that for most definitions of privacy, this is not the case. If you put two
<span class="math">\(k\)</span>-anonymous versions of the same data together, the result won't be
<span class="math">\(k\)</span>-anonymous. So if Alex and Brinn collaborate, they might be able to
reidentify users on their own… or even reconstruct all the original data! That's
not good news.</p>
<p>With differential privacy, you can avoid this failure mode. Suppose that you
gave differentially private data to Alex and Brinn. Each time, you used a
parameter of <span class="math">\(\varepsilon\)</span>. Then if they conspire, the resulting data is still
protected by differential privacy. The level of privacy is now weaker: the
parameter becomes <span class="math">\(2\varepsilon\)</span>. So they still gain some information, but you
can now quantify how much. This property is called <em>composition</em>.</p>
<p>This scenario sounds a bit far-fetched, but composition is super useful in
practice. Organizations often want to do many things with data. Publish
statistics, release an anonymized version, train machine learning algorithms…
Composition is a way to stay in control of the level of risk as new use cases
appear and processes evolve.</p>
<h1 id="conclusion">Conclusion</h1>
<p>I hope the basic intuition behind differential privacy is now clear. If you
remember a single thing, let this be this one-line summary: uncertainty in the
process means uncertainty for the attacker, which means better privacy.</p>
<p>I also hope that you're now wondering <em>how it actually works</em>! What hides behind
this magic that makes everything safe and private? Why does differential privacy
have all the awesome properties I've mentioned? This is the exact topic of the
<a href="differential-privacy-in-more-detail.html">next article</a> in this <a href="friendly-intro-to-differential-privacy.html">series</a>, which explains this in more detail while still
staying clear of heavy math.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:papers">
<p>The idea was first proposed in a
<a href="https://desfontain.es/PDFs/PhD/CalibratingNoiseToSensitivityInPrivateDataAnalysis.pdf">scientific paper<sup> (pdf)</sup></a> presented at <a href="https://www.iacr.org/workshops/tcc/">TCC</a> 2006, and can
also be found in a <a href="https://desfontain.es/PDFs/PhD/SelectivePrivacyGuarantees.pdf">patent<sup> (pdf)</sup></a> filed by Dwork and
McSherry in 2005. The name <em>differential privacy</em> seems to have appeared first
in an <a href="https://desfontain.es/PDFs/PhD/DifferentialPrivacy.pdf">invited paper<sup> (pdf)</sup></a> presented at <a href="http://eatcs.org/index.php/international-colloquium">ICALP</a> 2006 by
Dwork. <a class="footnote-backref" href="#fnref:papers" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>δ-presence, for when being in the dataset is sensitive2018-04-26T00:00:00+02:002018-04-26T00:00:00+02:00Damien Desfontainestag:desfontain.es,2018-04-26:/privacy/delta-presence.html<p><em>δ</em>-presence is a privacy notion which captures a different attack model than what we've previously seen. Let's understand why yet another definition is necessary, and what the solution looks like!</p><p><strong>Remember</strong> <a href="k-map.html"><span class="math">\(k\)</span>-map</a>? We used this definition when the attacker
didn't know who was in the dataset. Let's go back to this setting, with a
slightly different scenario. You're no longer a doctor studying human sexual
behavior. You're still a doctor, but this time, you're specialized in treating a
particular chronic disease. Instead of running a survey, you're running a
clinical trial for a new drug to treat this disease. Similarly, you want to
share the data with other people.</p>
<p>At first glance, these two settings look similar — but there is a crucial
difference. Which information is sensitive, exactly? For the survey, the
<em>answers</em> of each participant are sensitive, as they reveal intimate details.
But for the clinical study, <em>being</em> in the dataset is the sensitive information.
If someone figures out that you've taken part in the study, they learn that you
suffer from this disease.</p>
<p>So, what does it change in practice? Suppose that your dataset contains the
following records:</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">10</td>
</tr>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">12</td>
</tr>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">13</td>
</tr>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">13</td>
</tr>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">16</td>
</tr>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">43</td>
</tr>
</tbody>
</table>
<p>You do a little research on who lives in ZIP code 85535. You learn that in this
ZIP code: </p>
<ul>
<li><strong>5</strong> people have ages between 10 and 19;</li>
<li><strong>5</strong> people have ages between 20 and 29;</li>
<li><strong>10</strong> people have ages between 30 and 39;</li>
<li><strong>10</strong> people have ages between 40 and 49;</li>
<li>and <strong>20</strong> people are 50 or older.</li>
</ul>
<p>Transforming this part of your dataset to have it satisfy <span class="math">\(5\)</span>-map is easy:</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">10-19</td>
</tr>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">10-19</td>
</tr>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">10-19</td>
</tr>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">10-19</td>
</tr>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">10-19</td>
</tr>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">40-49</td>
</tr>
</tbody>
</table>
<p>… But what has gone wrong there?</p>
<p>An attacker, using only public data, knows that there are 5 people aged between
10 and 19 in ZIP code 85535. Then, by looking at your de-identified dataset, the
attacker can figure out that <em>all of them are part of your data</em>. Thus, they all
have this specific disease. The attacker learned something sensitive about
individuals, without re-identifying any record. Just like in the example of
<a href="l-diversity.html"><span class="math">\(l\)</span>-diversity</a>!</p>
<p>We need yet another definition. Introducing… <span class="math">\(\delta\)</span>-presence!</p>
<h1 id="definition">Definition</h1>
<p>Remember what we counted for our previous privacy definitions? For each
combination of quasi-identifier attributes:</p>
<ul>
<li>for <a href="k-anonymity.html"><span class="math">\(k\)</span>-anonymity</a>, we counted the number of records <em>in
the dataset</em>;</li>
<li>and for <a href="k-map.html"><span class="math">\(k\)</span>-map</a>, we counted the number of records <em>in the larger
population</em>.</li>
</ul>
<p>What went wrong in our leading example? For certain attributes, these numbers
were <em>equal</em>. To detect this, we now compute the <em>ratio</em> between those two
numbers. Then, the <span class="math">\(\delta\)</span> in <span class="math">\(\delta\)</span>-presence is the <em>largest</em> ratio across
the dataset.</p>
<p>Consider the dataset above. The ratio for the records (85535, 10-19) is <span class="math">\(5/5=1\)</span>,
and the ratio for the records (85535, 40-49) is <span class="math">\(1/10=0.1\)</span>. Thus, since we
defined <span class="math">\(\delta\)</span> as the greatest ratio, we have <span class="math">\(\delta=1\)</span>. Since the <span class="math">\(k\)</span> of
<span class="math">\(k\)</span>-map is always larger than the <span class="math">\(k\)</span> of <span class="math">\(k\)</span>-anonymity, this is the maximum
possible value of <span class="math">\(\delta\)</span>. Saying that a dataset satisfies <span class="math">\(1\)</span>-presence gives
zero guarantees.</p>
<p>Whether <span class="math">\(\delta=1\)</span> is not the only interesting thing. We also want this value to
be <em>small</em>. The lower, the better. Consider what it means if <span class="math">\(\delta=0.95\)</span>. The
attacker might learn that their target has a 95% chance of being in the dataset.
It's not quite a 100% certainty, but it still can be problematic. For example,
it might be more than enough for an insurance company to deny you coverage…</p>
<p>How do we get to a lower <span class="math">\(\delta\)</span> in our previous example? One solution would be
to generalize the age further:</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">10-39</td>
</tr>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">10-39</td>
</tr>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">10-39</td>
</tr>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">10-39</td>
</tr>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">10-39</td>
</tr>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">40-49</td>
</tr>
</tbody>
</table>
<p>Then, the ratio for the records (85535, 10-39) becomes <span class="math">\(5/(5+5+10)=0.25\)</span>. The
ratio for record (85535, 40-49) is still <span class="math">\(0.1\)</span>, so <span class="math">\(\delta=0.25\)</span>. (Assuming that
no other record in the dataset has ZIP code 85535, and all other records have a
smaller ratio).</p>
<p><span class="math">\(\delta\)</span>-presence was first proposed by Nergiz et al. in a <a href="https://desfontain.es/PDFs/PhD/HidingThePresenceOfIndividualsFromSharedDatabases.pdf">2007 paper<sup>
(pdf)</sup></a>. In this paper, the definition is a bit different. The
authors compute not only the largest ratio, but also the <em>smallest</em> one. The
<span class="math">\(\delta\)</span> parameter hides two parameters
<span class="math">\(\left(\delta_{\text{min}},\delta_{\text{max}}\right)\)</span>. This was done to protect
against the symmetric attack: hiding that someone is <em>not</em> in the dataset. I
never encountered a situation where this is a real concern, so I simplified it a
bit for this post.</p>
<h1 id="delta-presence-in-practice"><span class="math">\(\delta\)</span>-presence in practice</h1>
<p><span class="math">\(\delta\)</span>-presence is computed from the ratios between quantities used in
<span class="math">\(k\)</span>-anonymity and <span class="math">\(k\)</span>-map. While <span class="math">\(k\)</span>-anonymity is very easy to compute, <span class="math">\(k\)</span>-map
is <a href="k-map.html#and-now-some-practice">much harder</a>. As such, <span class="math">\(\delta\)</span>-presence has very similar practical
characteristics than <span class="math">\(k\)</span>-map. Since you don't typically have access to the full
larger dataset, you can't compute <span class="math">\(\delta\)</span> exactly. You can use a pessimistic
approximation if your data is a <a href="k-map.html#exception-1-secret-sample">sample</a> of a larger dataset that you
own. You can also do the work of estimating <span class="math">\(\delta\)</span>-presence <a href="k-map.html#exception-3-using-humans">by hand</a>.</p>
<p>What about statistical approximations? Nergiz et al. proposed an interesting
method in a <a href="https://desfontain.es/PDFs/PhD/DeltaPresenceWithoutCompleteWorldKnowledge.pdf">followup paper<sup> (pdf)</sup></a>. Unfortunately, two of
its requirements make it hardly usable in practical scenarios.</p>
<ul>
<li>First, to run the algorithm, you need to "describe your beliefs about the
world" (in a statistical sense). Unless you're a statistician, this is not
something you can really do.</li>
<li>Second, computing the algorithm exactly is very expensive. The authors propose
a lot of approximations to make it tractable… But then, using them makes the
results even more uncertain.</li>
</ul>
<p>Finally, if you still want to use this algorithm, you would also likely have to
implement it yourself. I don't know of any available software that does it for
you.</p>
<h1 id="conclusion">Conclusion</h1>
<p>Like <span class="math">\(k\)</span>-map, in theory, it often makes sense to use <span class="math">\(\delta\)</span>-presence. It's a
pity that both definitions are so difficult to use in practice! Having simpler
(and more usable) approximation algorithms would be great… Which is why I have
done some research work in that direction. And the results of this work will be
the topic of a future post! =)</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>l-diversity, because reidentification doesn't tell the whole story2018-02-19T00:00:00+01:002018-02-19T00:00:00+01:00Damien Desfontainestag:desfontain.es,2018-02-19:/privacy/l-diversity.html<p><em>l</em>-diversity is the first famous attempt at considering stronger attack models than simply reidentification attacks. Let's see how it works, and which flaws of <em>k</em>-anonmyity it fixes!</p><p><strong>Where</strong> does privacy risk come from, when releasing anonymized data? What
exactly can go wrong? Sweeney provided the first obvious answer: privacy risk
appears when you can <em>reidentify</em> a record. That makes sense, and led to the
definition of <a href="k-anonymity.html"><span class="math">\(k\)</span>-anonymity</a>. If you think your data is
anonymous, but somebody pinpoints a record and figures out who it is, clearly,
there's a problem.</p>
<p>But as researchers discovered shortly after, it's sometimes not enough. An
attacker might figure out private information about someone, <em>without
reidentifying their record</em>. Even if the dataset is <span class="math">\(k\)</span>-anonymous. How does this
magic work? First, we'll show how it works with an example, then we'll describe
the natural solution: <span class="math">\(l\)</span>-diversity.</p>
<div class="toc">
<ul>
<li><a href="#an-example">An example</a></li>
<li><a href="#l-diversity-the-obvious-fix">\(l\)-diversity: the obvious fix</a></li>
<li><a href="#wait-that-seems-too-easy">Wait, that seems too easy</a><ul>
<li><a href="#uncertain-information-can-still-be-sensitive">Uncertain information can still be sensitive</a></li>
<li><a href="#probabilistic-information-gain">Probabilistic information gain</a></li>
</ul>
</li>
<li><a href="#l-diversity-in-practice">\(l\)-diversity in practice</a><ul>
<li><a href="#the-good-news-implementation">The good news: implementation</a></li>
<li><a href="#the-bad-news-policy">The bad news: policy</a></li>
<li><a href="#the-other-bad-news-utility-loss">The other bad news: utility loss</a></li>
</ul>
</li>
<li><a href="#conclusion">Conclusion</a></li>
</ul>
</div>
<p></p>
<h1 id="an-example">An example</h1>
<p>Suppose you have the following database, which contains everyone in the country.</p>
<table>
<thead>
<tr>
<th style="text-align: center;">name</th>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
<th style="text-align: center;">diagnostic</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">Alice</td>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">34</td>
<td style="text-align: center;">Common flu</td>
</tr>
<tr>
<td style="text-align: center;">Bob</td>
<td style="text-align: center;">4212</td>
<td style="text-align: center;">39</td>
<td style="text-align: center;">Healthy</td>
</tr>
<tr>
<td style="text-align: center;">Camille</td>
<td style="text-align: center;">4732</td>
<td style="text-align: center;">39</td>
<td style="text-align: center;">Otitis</td>
</tr>
<tr>
<td style="text-align: center;">Dan</td>
<td style="text-align: center;">4743</td>
<td style="text-align: center;">23</td>
<td style="text-align: center;">Otitis</td>
</tr>
</tbody>
</table>
<p>(It's a rather small country.)</p>
<p>Now, you want to release an anonymized version of this database, for research
purposes. Following the <a href="k-anonymity.html"><span class="math">\(k\)</span>-anonymity</a> method, you start
by wondering which columns are identifying. Let's see.</p>
<ul>
<li><strong>name</strong> is obviously identifying: we have to remove it completely.</li>
<li><strong>ZIP code</strong> and <strong>age</strong> are <em>quasi-identifiers</em>. They can help you identify
someone, but reducing their precision might prevent this.</li>
<li><strong>diagnostic</strong> is <em>sensitive</em>, but since it's typically secret, we can
consider it non-identifying<sup id="fnref:identifying"><a class="footnote-ref" href="#fn:identifying">1</a></sup>.</li>
</ul>
<p>So, let's make this data <span class="math">\(k\)</span>-anonymous. Here, <span class="math">\(k=2\)</span>, because it's a small country.</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
<th style="text-align: center;">diagnostic</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">421*</td>
<td style="text-align: center;">30-39</td>
<td style="text-align: center;">Common flu</td>
</tr>
<tr>
<td style="text-align: center;">421*</td>
<td style="text-align: center;">30-39</td>
<td style="text-align: center;">Healthy</td>
</tr>
<tr>
<td style="text-align: center;">47**</td>
<td style="text-align: center;">20-39</td>
<td style="text-align: center;">Otitis</td>
</tr>
<tr>
<td style="text-align: center;">47**</td>
<td style="text-align: center;">20-39</td>
<td style="text-align: center;">Otitis</td>
</tr>
</tbody>
</table>
<p>Since all combinations of ZIP code & age appear twice, this data is
<span class="math">\(2\)</span>-anonymous. But now, suppose an attacker wants to find Camille's diagnostic.
The attacker knows that Camille has ZIP code 4732 and age 23. They can easily
figure out that Camille's record is the third or fourth one, but cannot know
which.</p>
<p>And there's the obvious problem: <em>both records have the same diagnostic</em>. So the
attacker can deduce that Camille's diagnostic is "Otitis". Even without knowing
which record is Camille's! <span class="math">\(k\)</span>-anonymity wasn't enough to protect Camille's
private information.</p>
<h1 id="l-diversity-the-obvious-fix"><span class="math">\(l\)</span>-diversity: the obvious fix</h1>
<p>So. Let's say that all users with the same quasi-identifier tuple are in the
same <em>bucket</em>. If all sensitive values are the same within a bucket, we might
leak private information. The obvious solution? Imposing some <em>diversity</em> in the
sensitive values associated to the same (generalized) tuple. </p>
<p>This is <span class="math">\(l\)</span>-diversity, as <a href="https://desfontain.es/PDFs/PhD/LDiversityPrivacyBeyondKAnonymity.pdf">introduced<sup> (pdf)</sup></a> in 2006 by
Machanavajjhala et al. It builds on the definition of <span class="math">\(k\)</span>-anonymity.
<span class="math">\(l\)</span>-diversity states that each bucket must have at least <span class="math">\(l\)</span> <em>distinct</em>
sensitive values. Of course, each bucket should contain at least <span class="math">\(l\)</span> users:
<span class="math">\(l\)</span>-diversity implies <span class="math">\(l\)</span>-anonymity.</p>
<p>Let's try to make the data above <span class="math">\(2\)</span>-diverse.</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
<th style="text-align: center;">diagnostic</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">4***</td>
<td style="text-align: center;">20-39</td>
<td style="text-align: center;">Common flu</td>
</tr>
<tr>
<td style="text-align: center;">4***</td>
<td style="text-align: center;">39</td>
<td style="text-align: center;">Healthy</td>
</tr>
<tr>
<td style="text-align: center;">4***</td>
<td style="text-align: center;">39</td>
<td style="text-align: center;">Otitis</td>
</tr>
<tr>
<td style="text-align: center;">4***</td>
<td style="text-align: center;">20-39</td>
<td style="text-align: center;">Otitis</td>
</tr>
</tbody>
</table>
<p>Now, consider our attacker from earlier, targeting Camille (third row). Like
before, the attacker is unable to know which records corresponds to Camille. But
besides, they also can't know whether Camille was healthy, or suffered from
otitis. The sensitive value stays private.</p>
<h1 id="wait-that-seems-too-easy">Wait, that seems too easy</h1>
<p>You might have noticed it immediately: the definition of <span class="math">\(l\)</span>-diversity has some
flaws. Let's list two of them.</p>
<h4 id="uncertain-information-can-still-be-sensitive">Uncertain information can still be sensitive</h4>
<p>What's the key idea behind <span class="math">\(l\)</span>-diversity? If the attacker has uncertainty over
the sensitive value, then we avoid leaking private info. But consider the
following database, which satisfies <span class="math">\(2\)</span>-diversity:</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
<th style="text-align: center;">diagnostic</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">42**</td>
<td style="text-align: center;">20-29</td>
<td style="text-align: center;">AIDS</td>
</tr>
<tr>
<td style="text-align: center;">42**</td>
<td style="text-align: center;">20-29</td>
<td style="text-align: center;">Hepatitis B</td>
</tr>
<tr>
<td style="text-align: center;">17**</td>
<td style="text-align: center;">30-39</td>
<td style="text-align: center;">Otitis</td>
</tr>
<tr>
<td style="text-align: center;">17**</td>
<td style="text-align: center;">30-39</td>
<td style="text-align: center;">Healthy</td>
</tr>
</tbody>
</table>
<p>Suppose the attacker knows that their target has ZIP code 4235 and age 25. The
target's record is one of the first two rows. The attacker can learn that their
target either has AIDS, or hepatitis B. They can't be sure <em>which one</em> is the
correct one… But they can infer that their target has a sexually transmitted
infection. This information, of course, might be embarrassing for the target!</p>
<p>How to fix this? One solution could be to group diagnostics into <em>categories</em>,
like <a href="https://www.findacode.com/code-set.php?set=ICD10CM">diagnostic code families</a>. Then, we can require that each bucket has
<span class="math">\(l\)</span> different categories of diagnostics. This way, the attacker can't
distinguish between STDs, external injuries, respiratory problems, etc.</p>
<p>Unfortunately, choosing these categories is a complicated policy question. There
are many possible combinations of sensitive values. Making sure that none of
them is sensitive sounds like a laborious task…</p>
<h4 id="probabilistic-information-gain">Probabilistic information gain</h4>
<p>Consider the following database, again satisfying <span class="math">\(2\)</span>-diversity:</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
<th style="text-align: center;">diagnostic</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">42**</td>
<td style="text-align: center;">20-29</td>
<td style="text-align: center;">Lupus</td>
</tr>
<tr>
<td style="text-align: center;">42**</td>
<td style="text-align: center;">20-29</td>
<td style="text-align: center;">Lupus</td>
</tr>
<tr>
<td style="text-align: center;">42**</td>
<td style="text-align: center;">20-29</td>
<td style="text-align: center;">Lupus</td>
</tr>
<tr>
<td style="text-align: center;">42**</td>
<td style="text-align: center;">20-29</td>
<td style="text-align: center;">Lupus</td>
</tr>
<tr>
<td style="text-align: center;">42**</td>
<td style="text-align: center;">20-29</td>
<td style="text-align: center;">Lupus</td>
</tr>
<tr>
<td style="text-align: center;">42**</td>
<td style="text-align: center;">20-29</td>
<td style="text-align: center;">Lupus</td>
</tr>
<tr>
<td style="text-align: center;">42**</td>
<td style="text-align: center;">20-29</td>
<td style="text-align: center;">Healthy</td>
</tr>
<tr>
<td style="text-align: center;">42**</td>
<td style="text-align: center;">20-29</td>
<td style="text-align: center;">Lupus</td>
</tr>
<tr>
<td style="text-align: center;">42**</td>
<td style="text-align: center;">20-29</td>
<td style="text-align: center;">Lupus</td>
</tr>
<tr>
<td style="text-align: center;">42**</td>
<td style="text-align: center;">20-29</td>
<td style="text-align: center;">Lupus</td>
</tr>
</tbody>
</table>
<p>Consider the same attacker as before: targeting someone with ZIP code 4235 and
age 25. They can't know their target's diagnostic for <em>certain</em>. But they can
get a strong <em>suspicion</em> that the target has lupus: 9 out of 10 records share
this diagnostic! An insurance company might increase someone's premium because
of a suspected pre-existing condition. Isn't that also a privacy issue?</p>
<p>How do we protect against this type of probabilistic information gain? Requiring
that sensitive attributes are <em>diverse</em> is not enough. We need to also require
that their <em>distribution</em> is roughly the same that the rest of the data. If 40%
of the records are "healthy" in the overall data, then each bucket must also
have roughly 40% of "healthy" records. This way, the attacker's knowledge can't
change <em>too much</em> from the baseline. This is the core idea behind another
definition named <span class="math">\(t\)</span>-closeness. I won't go into details here, but you can read
about it on <a href="https://en.wikipedia.org/wiki/T-closeness">Wikipedia</a> or in the <a href="http://desfontain.es/PDFs/PhD/TClosenessPrivacyBeyondKAnonymityAndLDiversity.pdf">original paper<sup>
(pdf)</sup></a> who introduced this idea.</p>
<p><small>Note: this idea is also relevant if the sensitive attribute is numeric,
like salary values. A yearly salary of €20,000 is very similar to €20,100:
applying <span class="math">\(l\)</span>-diversity doesn't make sense. By contrast, <span class="math">\(t\)</span>-closeness can
compare distributions in a more meaningful way.</small></p>
<h1 id="l-diversity-in-practice"><span class="math">\(l\)</span>-diversity in practice</h1>
<p>OK, so even with these flaws, how easy is it to use <span class="math">\(l\)</span>-diversity in practice?</p>
<h4 id="the-good-news-implementation">The good news: implementation</h4>
<p>From an algorithmic perspective, <span class="math">\(l\)</span>-diversity is very similar to <span class="math">\(k\)</span>-anonymity.
The basic blocks are the same: <a href="k-anonymity.html#building-block-1-generalization">generalization</a> and <a href="k-anonymity.html#building-block-2-suppression">suppression</a>. Finding the
best strategy is also done using trial-and-error heuristics. The approach used
for <span class="math">\(k\)</span>-anonymity is straightforward to adapt to <span class="math">\(l\)</span>-diversity.</p>
<p>Unsurprisingly, some software is available to implement it in practice. I won't
list them all here, but most <a href="k-anonymity.html#in-practice">options</a> introduced in my article about
<span class="math">\(k\)</span>-anonymity can also be used for <span class="math">\(l\)</span>-diversity.</p>
<h4 id="the-bad-news-policy">The bad news: policy</h4>
<p>Choosing the right value of <span class="math">\(k\)</span> for <span class="math">\(k\)</span>-anonymity is <a href="k-anonymity.html#how-to-choose-k">difficult</a>, but
<span class="math">\(l\)</span>-diversity is certainly not better. No official guideline or regulation will
help you choose the value of <span class="math">\(l\)</span>. And it's at least as hard to quantify the
"amount of privacy" obtained with a given choice of parameter.</p>
<p>Worse, the flaws described before mean that the question is even subtler than
that. Should we classify the sensitive values into categories? Impose that
sensitive values don't appear too often? If so, there are even more parameters
that one has to choose, and no good way to choose them.</p>
<h4 id="the-other-bad-news-utility-loss">The other bad news: utility loss</h4>
<p><span class="math">\(l\)</span>-diversity, despite its flaws, is strictly stronger than <span class="math">\(k\)</span>-anonymity. And
it should be relatively easy to use in practice, once we've chosen a policy…
Despite this, it is hardly ever used. A health data de-identification specialist
once told me that they only saw it in the wild a handful of times. By contrast,
using <span class="math">\(k\)</span>-anonymity is very common.</p>
<p>Why is that? I see two possible reasons.</p>
<p>First, the utility loss of <span class="math">\(l\)</span>-diversity is too significant compared to
<span class="math">\(k\)</span>-anonymity. A <a href="http://desfontain.es/PDFs/PhD/TheCostOfPrivacyDestructionOfDataMiningUtilityInAnonymizedDataPublishing.pdf">study<sup> (pdf)</sup></a> compared the utility loss of
different anonymization strategies. It found that applying <span class="math">\(3\)</span>-diversity dataset
was worse than using <span class="math">\(100\)</span>-anonymity! This particular result was for a
classification accuracy task, but you get the idea<sup id="fnref:examples"><a class="footnote-ref" href="#fn:examples">2</a></sup>.</p>
<p>Second, the privacy gains are not clear. Especially considering the flaws we
described above… And fixing those flaws hurts utility even more. With
<span class="math">\(t\)</span>-closeness, we hinder the ability to link demographic features with
diagnostics. But this type of analysis is exactly what healthcare researchers
want to do! If the privacy definition goes completely against this idea, it
won't get much love.</p>
<h1 id="conclusion">Conclusion</h1>
<p><span class="math">\(l\)</span>-diversity isn't a definition that definitely addresses a particular threat
model. Rather, it's a "fix" for one of <span class="math">\(k\)</span>-anonymity's most obvious flaws. But
in security, simply patching bugs one after the other isn't a great defense
mechanism... For privacy definitions too, fixing only one attack doesn't get you
very far.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:identifying">
<p>Which might be over-optimistic: some people make their health
issues public on social media, or the press can find out and publicize the
medical history of personalities. If you're actually doing this to anonymize a
real dataset, you should be more careful when classifying your columns. Here,
we're going with a simple assumption for the sake of simplicity. <a class="footnote-backref" href="#fnref:identifying" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:examples">
<p>It's hard to find many examples: negative results hardly ever get published… <a class="footnote-backref" href="#fnref:examples" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Book review: Crash Override2018-01-10T00:00:00+01:002018-01-10T00:00:00+01:00Damien Desfontainestag:desfontain.es,2018-01-10:/privacy/crash-override.html<p>A short review of <em>Crash Override</em>, by Zoë Quinn. tl;dr: you should read it, especially if you're building tech products or working in tech policy.</p><p><strong>Over the holidays</strong>, I read <a href="https://en.wikipedia.org/wiki/Zo%C3%AB_Quinn">Zoë Quinn</a>'s book, <em>Crash Override</em>. Zoë
Quinn is an independent games developer. After a gigantic harassment campaign
known as GamerGate targeted her, she became an activist against online hate. She
then funded an association to help other victims, the <a href="http://www.crashoverridenetwork.com/">Crash Override
Network</a>.
In this book, she tells the story of this huge campaign, and gives her thoughts
on how to prevent similar horrors from happening.</p>
<p><center>
<a href="https://www.amazon.com/dp/1610398084/"><img alt="Crash Override cover" src="https://desfontain.es/privacy/images/crash-override.jpg"></a>
</center></p>
<p>This book is eye-opening, well-written, and inspiring. Because of the difficult
topic, it's not an easy read. She tells her story in a very personal way,
describing what she went through when thousands of trolls were harassing her and
her close ones. The book is worth a read just for this testimony. It slaps you
in the face, reminding you that what happens online is as real as what happens
in the physical world. It gives you a healthy dose of empathy towards people you
interact with on the Internet. Even if you're not an harasser, it's so easy to
forget the actual person behind the nickname or the avatar…</p>
<p>This intense dose of empathy is critically valuable if you build tech products,
or work in tech policy. Quinn describes everything that went wrong not only with
the humans, but also with the technology. It's horrifying and infuriating.
Popular tech products and social networks simply suck at dealing with abuse and
harassment on their platforms. Quinn is doing an excellent job at analyzing
their flaws, and detailing what can be explained by cluelessness and what is the
sign of truly rotten ideologies. If you're working in tech, you really really
<em>really</em> should learn about these issues. Go read this book.</p>
<p>It's chilling to read Quinn's descriptions of the mass movements behind
GamerGate. It serves as a good reminder of how powerful and devastating group
dynamics can be. Some of the things harassers did sound completely unreal. Quinn
has to explain her situation to many people, and they often don't understand nor
accept it: it sounds so <em>ridiculous</em>. It's difficult to believe actual humans
would harass people this way. Especially people they've never met, in such a
violent and persistent way.</p>
<p>But online trolls don't see their target as human, either. Instead, victims are
seen as an abstract concept of "evil". Abusers think that everything they do to
their victims is well-deserved. And the more they fight evil, the more feel like
"heroes", fighters on the good side in the grand scheme of things. This is
nothing new: anti-immigration rhetoric is a classical example of this
phenomenon. Nonetheless, GamerGate really was the symptom of something
profoundly rotten in parts of society. Victims realized that, of course, but
nobody listened to them…</p>
<p>After talking about GamerGate, Quinn explains how to efficiently fight online
hate. Given her personal experience and her work as an activist, she's uniquely
qualified. She developed a set of principles, and offers concrete advice both
for victims, bystanders, as well as technologists.</p>
<p>The technical advice for victims (or for people potentially at risk, which
really means everyone) is sound, but doesn't get into much detail. If that's
what you're looking for, consider reading <a href="https://www.amazon.com/dp/B013HA1V4S/">The Smart Girl's Guide to
Privacy</a> (by <a href="https://en.wikipedia.org/wiki/Violet_Blue">Violet Blue</a>). You can also check out the
<a href="http://www.crashoverridenetwork.com/resources.html">resources</a> on the Crash Override's Network website.</p>
<p>The "human" advice, on the other end, is quite complete and excellent.
Recommendations are concrete and sound (typically not "just go offline,
stupid"). Many common reactions, both for victims and bystanders, are
counter-productive. Quinn does an excellent job at detailing these and
explaining what to do instead.</p>
<p>The end of the book surprised me. Quinn writes a whole section about empathy
towards harassers and abusers. She explains that there was a time where <em>she</em>
was an angry troll, posting nasty comments to people she didn't know online.
Like her own abusers, she was doing this for very normal and human reasons.
Status, positive feedback from your peers, technical prowess, acceptance within
a social group… Even if the result is incredibly evil, this is very <em>normal</em>
from a psychological perspective. Especially now, with all the gamification (and
financial rewards!) of social networks.</p>
<p>This last part of her testimony really resonated with me. I was the same when I
was younger. I browsed 4chan regularly. I was only a <a href="https://en.wikipedia.org/wiki/Lurker">lurker</a>: my
English was bad, so I was afraid of people mocking me. Nonetheless, I found
comfort in the nihilism, the irreverence, and the sense that it was okay to be
different and weird. I almost never talked to anyone, but I still felt like
these were people who could understand me. All the racism, sexism, general
awfulness… Even that felt okay. It was <em>obviously</em> ironic<sup id="fnref:irony"><a class="footnote-ref" href="#fn:irony">1</a></sup>, and there was
also a lot of self-deprecating humor. But mostly, laughing at everything just
came with the nihilism. Putting groups of people I felt like I didn't understand
(especially women) in "enemy" boxes was very, very easy.</p>
<p>Then, my life got better and I progressively stopped going there. But reading
Quinn's own account of being an online troll made me wonder: would I have taken
part in GamerGate, had it been earlier? If I had had just a little bit more
bottled-up anger and awkwardness? If my English had been better? I certainly
wouldn't have done the right thing — even as a shy lurker, I didn't do anything
right back then. This was a harsh (and unexpected) realization. For me, it was a
valuable take-away. If you partly recognize yourself in what I clumsily tried to
describe there, I would also advise reading this book.</p>
<p>Anyway, she ends the book saying that more abuse or violence towards harassers
aren't going to make the hate go away. Instead, Quinn encourages speaking up in
a non-confrontational way, listening and empathizing… After everything she went
through, this is nothing short of inspiring!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:irony">
<p>It was not. I was extremely dumb, and extremely privileged, so I don't
think I realized that. Or, more probably, I chose not to. Not seeing how wrong
it was, not speaking up… This enabled real and horrific violence. I regret
this time a lot, and feel ashamed when thinking of it, but there is no excuse
for any of it. <a class="footnote-backref" href="#fnref:irony" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>k-map, the weird cousin of k-anonymity2017-10-10T00:00:00+02:002018-04-26T00:00:00+02:00Damien Desfontainestag:desfontain.es,2017-10-10:/privacy/k-map.html<p>Weakening <em>k</em>-anonymity, really? This sounds weird, but this can actually be quite reasonable. Let's learn why!</p><p><strong>Suppose</strong> that you're a doctor who studies human sexual behavior. You want to
run a study with all the patients that you can find, but you don't find a lot of
volunteers. You only end up with about 40 subjects.</p>
<p>After you've ran your study and collected data, you want to share this data with
other researchers. You look at the attributes, and deduce that ZIP code and age
are likely to be used in reidentification attacks. To share it in a safe way,
you're thinking of <a href="k-anonymity.html"><span class="math">\(k\)</span>-anonymity</a>.</p>
<p>When trying to find a strategy to obtain <span class="math">\(k\)</span>-anonymity, you find out that you
would have to lose a lot of information. For <span class="math">\(k=10\)</span>, a rather small value, you
end up with buckets like <span class="math">\(20\le age\lt 50\)</span>. That makes sense: you have only few
people in your database, so you have to bundle together very different age
values.</p>
<p>But when you think about it, you start questioning whether you really need
<span class="math">\(k\)</span>-anonymity. Who are the attackers, in your scenario? The researchers with
whom you share the data, and possibly unknown parties if the data ever leaks.
None of these people have background information about who is in the dataset.
Thus, the attacker doesn't just have to distinguish between different records,
but to actually find the <em>real identity</em> of a record based on its information.
This attacker has significantly weaker capabilities than for <span class="math">\(k\)</span>-anonymity!</p>
<p>Let's look at two different rows in this database.</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">79</td>
</tr>
<tr>
<td style="text-align: center;">60629</td>
<td style="text-align: center;">42</td>
</tr>
</tbody>
</table>
<p>At first glance, the amount of information for this two individuals seems to be
the same. But let's take a look at the values…</p>
<ul>
<li><a href="https://www.unitedstateszipcodes.org/85535/">85535</a> corresponds to a place in Arizona named Eden. Approximately 20 people
live in this ZIP code. How many people do you think are exactly 79 years old
in this particular ZIP code? Probably only one.</li>
<li><a href="https://www.unitedstateszipcodes.org/60629/">60629</a> corresponds to a part of the Chicago metropolitan area. More than
100,000 people live there. How many of them are 42 years old? A thousand, at
least, and probably more!</li>
</ul>
<p>It seems that it would be very easy to reidentify the first row, but that we
don't have enough information to reidentify the second row. But according to
<span class="math">\(k\)</span>-anonymity, both rows might be completely unique in the dataset.</p>
<p>Obviously, <span class="math">\(k\)</span>-anonymity doesn't fit this use case. We need a different
definition: that's where <span class="math">\(k\)</span>-map comes in.</p>
<h1 id="definition">Definition</h1>
<p>Just like <a href="k-anonymity.html"><span class="math">\(k\)</span>-anonymity</a>, <span class="math">\(k\)</span>-map requires you to determine
which columns of your database are <em>quasi-identifiers</em>. This answers the
question: what can your attacker use to reidentify their target?</p>
<p>But this information alone is not enough to compute <span class="math">\(k\)</span>-map. In the example
above, we assumed that the attacker doesn't know whether their target is in the
dataset. So what are they comparing a given row with? With all other individuals
sharing the same values in a larger, sometimes implicit, dataset. For the
previous example, this could be "everybody living in the US", if you assume the
attacker has no idea who could have this genetic disease. Let's call this larger
table the <em>reidentification dataset</em>.</p>
<p>Once you picked the quasi-identifiers and the reidentification dataset, the
definition is straightforward. Your data satisfies <span class="math">\(k\)</span>-map if every combination
of values for the quasi-identifiers appears at least <span class="math">\(k\)</span> times <em>in the
reidentification dataset</em>.</p>
<p>In our example, this corresponds to counting the number of people in the US who
share the quasi-identifier values of each row in your dataset. Consider our tiny
dataset above:</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">85535</td>
<td style="text-align: center;">79</td>
</tr>
<tr>
<td style="text-align: center;">60629</td>
<td style="text-align: center;">42</td>
</tr>
</tbody>
</table>
<p>We said earlier than the values of the first row matched only one person in the
US. Thus, this dataset does not satisfy <span class="math">\(k\)</span>-map for any value of <span class="math">\(k\ge 2\)</span>.</p>
<p>How do we get a larger <span class="math">\(k\)</span>? We could generalize the first value like this:</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">85***</td>
<td style="text-align: center;">79</td>
</tr>
<tr>
<td style="text-align: center;">60629</td>
<td style="text-align: center;">42</td>
</tr>
</tbody>
</table>
<p>ZIP codes between 85000 and 85999 include the entire city of <a href="https://en.wikipedia.org/wiki/Phoenix,_Arizona">Phoenix</a>. There
are 36,000+ people between 75 and 84 years old in Phoenix, according to some
<a href="http://phoenix.areaconnect.com/statistics.htm">old stats</a>. It's probably safe to assume that there are more than 1,000 people
who match the quasi-identifiers values of the first row. We saw earlier that the
second row also matched 1,000+ people. So this generalized dataset satisfies
1000-map.</p>
<h1 id="attack-model-considerations">Attack model considerations</h1>
<p>Wait a second, why does this feel like cheating? What happened there, to give us
such a generous number so easily? This comes from the generous assumptions we
made in our attack model. We assumed that the attacker had <em>zero</em> information on
their target, except that they live in the US (which is implied by the presence
of ZIP codes). And with only the information (ZIP code, age), you don't need a
lot of generalization to make each row of your dataset blend in a large crowd.</p>
<p>To make this attack model stronger, you could assume that the attacker will use
a <em>smaller</em> reidentification database. For example, suppose that your genetic
disease you're studying requires regular hospital check-ups. The attacker could
restrict their search only to people who have visited a hospital in the last
year. The number of possible "suspects" for each value tuple gets smaller, so
the <span class="math">\(k\)</span> of <span class="math">\(k\)</span>-map decreases too<sup id="fnref:generic"><a class="footnote-ref" href="#fn:generic">1</a></sup>.</p>
<p><span class="math">\(k\)</span>-map is inherently a <em>weak</em> model. So when choosing the quasi-identifiers and
reidentification dataset, you have to think hard at what an attacker could do.
If your attacker doesn't have lots of resources, it can be reasonable to assume
that they won't get more data than, say, the voter files from your state. But if
they can figure out more about your users, and you don't really know which
reidentification dataset they could use, maybe <span class="math">\(k\)</span>-anonymity is a safer
bet<sup id="fnref:safer"><a class="footnote-ref" href="#fn:safer">2</a></sup>.</p>
<h1 id="and-now-some-practice">And now, some practice</h1>
<p>OK, enough theory. Let's learn how to compute <span class="math">\(k\)</span>-map in practice, and anonymize
your datasets to make them verify the definition!</p>
<p>… There's one slight problem, though.</p>
<p>It's usually impossible.</p>
<p>Choosing the reidentification dataset is already a difficult exercise. Maybe you
can afford to make generous assumptions, and assume the attacker doesn't know
much. At best, you think, they'll buy voter files, or a commercial database,
which contains everyone in your state, or in the US. But… then what?</p>
<p>To compute the maximum <span class="math">\(k\)</span> such as your dataset verifies <span class="math">\(k\)</span>-map, you would
first need to get the reidentification dataset yourself. But commercial
databases are expensive. Voter files might not be legal for you to obtain (even
though an evil attacker could break the law to get them).</p>
<p>So, most of the time, you can't actually check whether your data satisfies
<span class="math">\(k\)</span>-map. If it's impossible to check, it's also impossible to know exactly which
strategy to adopt to make your dataset verify the definition.</p>
<h4 id="exception-1-secret-sample">Exception 1: secret sample</h4>
<p>Suppose you're not releasing all your data, but only a <em>subset</em> (or <em>sample</em>) of
a bigger dataset that you own. Then, you can compute the <span class="math">\(k\)</span>-map value of the
sample with regard to the original, bigger dataset. In this case, choosing
<span class="math">\(k\)</span>-map over <span class="math">\(k\)</span>-anonymity is relatively safe.</p>
<p>Indeed, your original dataset is certainly <em>smaller</em> than the reidentification
dataset used by the attacker. Using the same argument as above, this means that
you will obtain a <em>lower bound</em> on the value of <span class="math">\(k\)</span>. Essentially, you're being
pessimistic, which means that you're on the safe side.</p>
<p>Even if the attacker has access to the original dataset, they won't know which
records are in the sample. So if the original dataset is secret, or if you've
chosen the sample in a secret way, <span class="math">\(k\)</span>-map is a reasonable definition to use,
and you can compute a pessimistic approximation of it.</p>
<h4 id="exception-2-representative-distribution">Exception 2: representative distribution</h4>
<p>This case is slightly different. Suppose that you can make the assumption that
your data is a <a href="http://arx.deidentifier.org/anonymization-tool/configuration/#a27"><em>representative</em></a> (or <em>unbiaised</em>) sample of a larger
dataset. This might be a good approximation if you selected people (uniformly)
at random to build your dataset, or if it was gathered by a polling
organization.</p>
<p>In this case, you can compute an estimate of the <span class="math">\(k\)</span>-map value for your data,
even without the reidentification dataset. The statistical properties which
enable this, and the methods you can use, are pretty complicated: I won't
explain them in detail here. They are mentioned and compared in <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2528029/">this
paper</a>, which has references to the original versions of each of them.</p>
<h4 id="exception-3-using-humans">Exception 3: using humans</h4>
<p>For the case of our doctor earlier, if the dataset is small enough, a motivated
data owner could actually do the job of an attacker "by hand". Go through each
record, and try to map it to a real person, or estimate the chances of it being
possible. We pretty much did that in this article!</p>
<p>This is very approximative, and obviously not scalable. But for our imaginary
doctor, it might be a reasonable solution!</p>
<h4 id="implementations">Implementations</h4>
<p><a href="http://arx.deidentifier.org/">ARX</a> implements the methods from exceptions 1 and 2. Documentation for the
first one can be found <a href="http://arx.deidentifier.org/anonymization-tool/configuration/#a27">here</a>. Instructions to estimate the number of
<em>unique</em> values assuming uniformity can be found <a href="http://arx.deidentifier.org/anonymization-tool/risk-analysis/#a56">here</a>. Originally,
<a href="http://neon.vb.cbs.nl/casc/..%5Ccasc%5Cmu.htm">μ-ARGUS</a> was the first software with this feature, but I couldn't run
it on my machine, so I can't say much about it.</p>
<h1 id="conclusion">Conclusion</h1>
<p>You might wonder why I wrote an entire article on a definition that is hardly
used because of how impractical it is. In addition to the unique problems that
we talked about in this article, the limitations of <span class="math">\(k\)</span>-anonymity also apply.
It's difficult to choose <span class="math">\(k\)</span>, non-trivial to pick the quasi-identifiers, and
even trickier to model the reidentification database.</p>
<p>The definition also didn't get a lot of attention from academics. Historically,
<span class="math">\(k\)</span>-anonymity came first<sup id="fnref:history"><a class="footnote-ref" href="#fn:history">4</a></sup>. Then, people showed that <span class="math">\(k\)</span>-anonymity was
sometimes not sufficient to protect sensitive data, and tried to find <em>stronger</em>
definitions to fix it. Weaker definitions were, of course, less interesting.</p>
<p>Nonetheless, I find that it's an interesting relaxation of <span class="math">\(k\)</span>-anonymity. It
shows one of its implicit assumptions: the attacker knows that their target
belongs to the dataset. This assumption is sometimes too pessimistic: it might
be worth considering alternate definitions.</p>
<p>Choosing a privacy model is all about modeling the attacker correctly. Learning
to question implicit assumptions can only help!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:generic">
<p>There is a generic version of this argument. Let's call your
database <span class="math">\(D\)</span>, and suppose <span class="math">\(R\)</span> and <span class="math">\(R^\prime\)</span> are two possible reidentification
databases. Suppose that <span class="math">\(R^\prime\)</span> is "larger" than <span class="math">\(R\)</span> (each element of <span class="math">\(R\)</span>
appears in <span class="math">\(R^\prime\)</span>). Then if <span class="math">\(D\)</span> satisfies <span class="math">\(k\)</span>-map with regard to <span class="math">\(R\)</span>, it
also satisfies <span class="math">\(k\)</span>-map with regard to <span class="math">\(R^\prime\)</span>. The reverse is not true. <a class="footnote-backref" href="#fnref:generic" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:safer">
<p>One simple consequence of the previous footnote is that if a dataset
<span class="math">\(D\)</span> verifies <span class="math">\(k\)</span>-anonymity, then it automatically verifies <span class="math">\(k\)</span>-map for any
reidentification dataset<sup id="fnref:assumption"><a class="footnote-ref" href="#fn:assumption">3</a></sup>. <a class="footnote-backref" href="#fnref:safer" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:assumption">
<p>I didn't say this explicitly, but the reidentification dataset is
always assumed to contain all rows from your dataset. It's usually not the
case in practice because data is messy, but it's a safe assumption. Hoping
that your attacker will just ignore some records in your data would be a bit
overly optimistic. <a class="footnote-backref" href="#fnref:assumption" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:history">
<p>Latanya Sweeney first mentioned the idea behind <span class="math">\(k\)</span>-map in
<a href="https://desfontain.es/PDFs/PhD/AchievingKAnonymityPrivacyProtectionUsingGeneralizationAndSuppression.pdf">this 2002 paper<sup> (pdf)</sup></a>, several years after the
introduction of <span class="math">\(k\)</span>-anonymity. <a class="footnote-backref" href="#fnref:history" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Book review: Twitter and Tear Gas2017-10-09T00:00:00+02:002017-10-09T00:00:00+02:00Damien Desfontainestag:desfontain.es,2017-10-09:/privacy/twitter-tear-gas.html<p>A short review of <em>Twitter and Tear Gas: The Power and Fragility of Networked Protest</em>, by Zeynep Tufekci. tl;dr: you should read it, especially if you participate in activist movements.</p><p><strong>I recently finished</strong> reading <em>Twitter and Tear Gas: The Power and Fragility
of Networked Protest</em>, by <a href="https://en.wikipedia.org/wiki/Zeynep_Tufekci">Zeynep Tufekci</a>. It's a long yet dense essay
on how modern protests work, and why they sometimes don't. Tufekci has a long
experience as an activist in many different protests around the world. She also
has a strong education in technology and in social sciences, and her work
focuses on the intersection between the two. In short, she is the perfect person
to write a book conceptualizing modern protests and their use of technology.
Unsurprisingly, the essay makes for a fascinating and enlightening read.</p>
<p><center>
<a href="https://www.twitterandteargas.org/"><img alt="Twitter and Tear Gas cover" src="https://desfontain.es/privacy/images/twitter-tear-gas.png"></a>
</center></p>
<p>Here's an example. What does it mean when many people march in the streets? It
displays power: the power to send the word out, to convince people to join, to
organize logistics. But the <em>actual march</em> isn't scary to people in power: the
<em>implications</em> are. If an organization is able to gather many people for a
march, then this movement is capable of other things. Boycotts, strikes,
fundraisers for your political opponents, influence in the media…</p>
<p>All those things <em>actually</em> cause headaches to politicians, and make change more
likely. The protest itself merely serves as a signal. A few decades ago, it was
a <em>strong</em> signal: only very powerful movements could put a large number of
people on the streets. So if you could pull off a large protest, it meant that
your movement could do all those other annoying things. Social media and
technological tools change this. With them, it's much easier to plan an event,
get the word out, and have many people rally around a cause for an afternoon.
This should be good news for protesters… Except it also means that large
protests are no longer such a show of strength. "Easier" also means "less
impressive". And the people in power have understood this.</p>
<p>Consider movements such as the anti-war demonstrations of the Bush era, Occupy,
or the more recent Woman's March. Politicians were able to pretty much ignore
protesters: once everyone gets home, nothing happens. Only the most motivated of
political opponents might cause <em>actual</em> issues later on. Worse, their number is
not directly related to the size of the protest itself. So the protest can be
very impressive (especially when comparing it with historical protests), and
still not scare anyone in power.</p>
<p>I picked this particular insight to try and convince you to read the book… But
that's obviously only a tiny part of what is there. Tufekci provides simple
concepts to understand how tech interacts with social movements. It's rigorous,
detailed, and illustrated with plenty of historical examples. The author doesn't
assume you know these examples already (even for "famous" events, like civil
rights movements in the US). This is great for people like me with a limited
knowledge of history ^^</p>
<p><em>Twitter and Tear Gas</em> doesn't only evoke protests. At the intersection between
tech and social movements, there are also misinformation campaigns, online
harassment, social network policies and their consequences… Each of those is
discussed in the book, always with the same academic rigor, lively examples, and
clear writing.</p>
<p>The book is an excellent read from an intellectual perspective: it made many
ideas <em>clearer</em> and <em>simpler to understand</em> for me. This feeling is the best
indicator I know of good science! But you can also read this book as an
instruction manual. How to build "muscle" for a movement, how to orient it
towards the most efficients means of action, how to deal with misinformation and
censorship… Using the technological tools that were developed in the last few
decades.</p>
<p>Everyone working in tech could probably benefit from reading <em>Twitter and Tear
Gas</em>. If you're an activist, I'd say it's pretty much required reading. Go buy
it <a href="https://www.amazon.com/Twitter-Tear-Gas-Fragility-Networked/dp/0300215126/">there</a> or <a href="http://yalebooks.com/book/9780300215120/twitter-and-tear-gas">there</a>, or if you can't afford it, <a href="https://www.twitterandteargas.org/downloads/twitter-and-tear-gas-by-zeynep-tufekci.pdf">download it for
free</a>: it's licensed under <a href="http://technosociology.org/?p=1751">Creative Commons</a>! (This excellent model of
publishing alone is a good reason to buy the book.)</p>Biometrics: authentication or identification?2017-09-27T00:00:00+02:002017-09-27T00:00:00+02:00Damien Desfontainestag:desfontain.es,2017-09-27:/privacy/authentication-vs-identification.html<p>Know the difference. It probably won't save your life, but it can certainly avoid you saying nonsensical things on the Internet.</p><p><strong>Earlier this month</strong>, there was lots of chatter online about the new iPhone's
FaceID feature: it allows you to unlock your device just by looking at it.
Behind the scenes are some hardware and algorithms which create a 3D map of your
face, and determine whether you're the phone's rightful owner.</p>
<p>Many people seemed to not understand the difference between <em>authentication</em> and
<em>identification</em>. Both authentication and identification can use biometric data,
like facial recognition. Nonetheless, these use cases are fundamentally
different. I'll try to explain why — I hope this can enlighten the debate around
features like this a little bit.</p>
<h1 id="authentication">Authentication</h1>
<p>Authentication is what you do when you log in to some Internet service, or when
you unlock your phone. First, you <em>announce your identity</em> to the authentication
system (e.g. a log-in page or lock screen). Then, you try to <em>prove</em> to the
system that you're indeed who you pretend to be. For an Internet service,
identity can mean your e-mail address. For a phone, it's more implicit: you're
trying to prove you're the owner of the phone.</p>
<p>The attack model is the following: some evil person <em>pretends to be you</em>, and
tries to prove it to the authentication system to get access to your data. This
attacker can be of various types:</p>
<ul>
<li>an abusive partner who wants to look into your phone,</li>
<li>a scammer who wants to steal your identity,</li>
<li>a spy who wants to penetrate a company's network…</li>
</ul>
<p>Fundamentally, authentication protects against <em>unauthorized access to data</em>.</p>
<h1 id="identification">Identification</h1>
<p>Identification is trying to figure out <em>who someone is</em> based on some
characteristics they have, or data they produced. It's what the police does when
running a fingerprint against a database of suspects. It's what privacy
researchers do when they try to show that a data release has not been <a href="k-anonymity.html">properly
anonymized</a>.</p>
<p>The attack model here is that somebody tries to <em>find your identity</em>. To
succeed, an attacker needs to have a <em>list of suspects</em>, and enough information
to <em>distinguish</em> who you are among all possibilities.</p>
<h1 id="good-authentication-vector-vs-good-identification-vector">Good authentication vector vs. good identification vector</h1>
<p>From these different attack models, a first distinction emerges.</p>
<ul>
<li>If a piece of data is <em>secret</em>, it will work well as an authentication vector.
Passwords, codes embedded in security keys, or one-time SMS codes, are classic
examples.</li>
<li>If a piece of data is <em>public</em>, or at least known to the attacker, it can work
as an identification vector. Names, dates of birth or phone numbers are good
candidates.</li>
</ul>
<p>A second distinction is on the <em>amount</em> of information present in the data.</p>
<ul>
<li>To authenticate someone, you don't always need lots of info. For example, a
4-digit PIN code is enough to get a decent security on a phone, provided only
a few retries are allowed.</li>
<li>To identify someone, you need more than this. Even if you somehow get your
hands on a database which contains everyone's PIN code, each one would
correspond to many people. A PIN code alone wouldn't be enough: you need some
context or more data to reliably identify someone. <br><br></li>
</ul>
<h1 id="biometrics-for-authentication">Biometrics for authentication</h1>
<p>Biometrics seem to be blurring the line. Fingerprints are not exactly secret,
right? Your face is probably also all over social media. So how come they are
more and more used as authentication methods?</p>
<p>It turns out that the <em>secretness</em> of authentication data is not a required
property. All we need is <em>unforgeability</em>: an attacker must not be able to
impersonate you. If a secret is well-protected, it's difficult to falsify: the
attacker can't imitate what they don't know. But biometric info can be quite
unforgeable, even if it's not technically secret. It's easy to find what
someone's face or fingerprint looks like, but it's hard to create a fake version
of it.</p>
<p>Some folks have written excellent articles on the difficulty of bypassing
biometric authentication. So, instead of diving into the details, I'll simply
recommend <a href="https://www.troyhunt.com/face-id-touch-id-pins-no-id-and-pragmatic-security/">this excellent
post</a>
from Troy Hunt's blog.</p>
<!-- -->
<h1 id="biometrics-for-identification">Biometrics for identification</h1>
<p>Information being <em>public</em> doesn't mean that there exists a central database
containing everyone's data. This is especially true for biometric info. Most
attackers don't have access to global fingerprint or facial recognition
databases (yet)… But when they do, it definitely raises serious privacy
concerns. </p>
<p>Classic identification attacks focus on finding the person behind a pseudonym or
identifier. Identifiers can be phone numbers, e-mail addresses… Over time, you
can change pseudonyms and identifiers<sup id="fnref:changed"><a class="footnote-ref" href="#fn:changed">1</a></sup>. You can also maintain separate
identities, for example when you use a different email address for services you
don't trust.</p>
<p>Biometric identification doesn't have these nice properties. You can't change
your face or your fingerprints! And you can't use a different right thumb with
border agents of different countries, either.</p>
<p>Furthermore, you also have less <em>control</em> over your biometric information. You
can decide not to interact with a given online service if you don't trust it.
But if you're living a "normal" life in a Western city, your face will most
certainly be caught and recorded by many surveillance cameras.</p>
<p>Creating a facial recognition database is becoming simpler and cheaper. In
Russia, pro-Putin activists identified anti-government protestors using pictures
gathered from social media<sup id="fnref:findface"><a class="footnote-ref" href="#fn:findface">2</a></sup>. "Researchers" are creating algorithms to
detect sexual orientation<sup id="fnref:orientation"><a class="footnote-ref" href="#fn:orientation">3</a></sup> or gender identity<sup id="fnref:gender"><a class="footnote-ref" href="#fn:gender">4</a></sup>. They used
data from dating apps or video sharing services, and didn't ask anyone for
consent.</p>
<p>Using biometric data for identification is not inherently problematic. For
example, it helps catching violent criminals. Yet, the privacy concerns are most
definitely justified.</p>
<h1 id="are-those-really-distinct-problems">Are those really distinct problems?</h1>
<p>So, biometric identification is creepy, but biometric authentication isn't
always problematic. But wait. If people build biometric authentication systems…
How do they recognize someone's face or fingerprint if they don't store it
somewhere? Didn't the engineers behind FaceID had to build a biometric database?
Couldn't evil people use this for identification? </p>
<p>Not necessarily. For many of those tools, it is a specific design goal to <em>not</em>
make biometric identification easier. This is achieved through a series of risk
mitigation mechanisms<sup id="fnref:faceid"><a class="footnote-ref" href="#fn:faceid">5</a></sup>:</p>
<ul>
<li>The biometric data exists only on the user's phone, not in a central place.
The phone vendor doesn't need to unlock your phone! So it doesn't need this
information. The database doesn't exist in the first place.</li>
<li>The data lives in a specific piece of hardware called "Secure Enclave". This
chip encrypts and stores secrets <em>independently</em> of other parts of the phone.
Even if a hacker takes control of your phone, or a thief steals it, they can't
read the biometric data stored on it. Building a biometric database from
hacked iPhones is near-impossible.</li>
<li>Pictures taken during authentication are immediately discarded. Only the
pictures used for <em>enrollment</em> (when you set up FaceID) are stored. This way,
you know what is stored on your phone. This way, the chip doesn't store
pictures that you wouldn't want stored there.</li>
</ul>
<p>In addition, some fingerprint systems store only <em>partial</em> information on their
users. Remember how a 4-digit PIN was enough for certain authentication systems?
Similarly, partial biometric data can be enough to be a good authentication
vector. So even if data leaks, the exact info might not be enough to uniquely
identify someone.</p>
<p>Authentication is a <em>different problem</em> than identification. Thus, a system
designed for the former can also mitigate risk against the latter.</p>
<p>Does this mean we shouldn't worry about biometric authentication systems? Ha!
No.</p>
<h4 id="point-of-failure-1-the-tech">Point of failure 1: the tech</h4>
<p>I'm quite confident that Apple's new FaceID system is reasonably secure.
Zero-days vulnerabilities for iOS are worth
<a href="https://www.zerodium.com/program.html">millions</a>. That's a good sign that Apple
has a strong security team who know what they're doing.</p>
<p>But there should be a <em>lot</em> of healthy skepticism when anyone introduces a new
system like this. Data breaches happen all the time. If a biometric
authentication system is badly designed, the potential consequences are
catastrophic.</p>
<h4 id="point-of-failure-2-the-people">Point of failure 2: the people</h4>
<p>Did I convince you that authentication and identification are not the same
thing? Excellent. Will most people understand the distinction? I'm not exactly
optimistic =(</p>
<p>FaceID will probably make people more comfortable with facial recognition
itself. And if the technology gets normalized, this will lead to more
problematic uses being more easily accepted.</p>
<p>This week, I heard about future plans for the London public transportation
system. They are considering <a href="https://www.wired.co.uk/article/train-station-face-recognition-gateless-gate-technology">facial recognition</a> as a replacement for
magnetic cards containing tickets. Have your face recognized when you enter and
leave the subway, get charged later. This is an <em>identification</em> system. The
privacy implications are vastly different, and the consequences of security
incidents could be catastrophic.</p>
<p>Will people understand the difference?</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:changed">
<p>Are you thinking "wait a second, I can't change my social security
number…"? Excellent point! This is one of the many reasons why SSNs make such
terrible identifiers. <a class="footnote-backref" href="#fnref:changed" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:findface">
<p>Here's a scary
<a href="https://www.theguardian.com/technology/2016/may/17/findface-face-recognition-app-end-public-anonymity-vkontakte">article</a>
about this thing. Their success rate was pretty terrible, but this didn't stop
them. And the tech is getting better fast. <a class="footnote-backref" href="#fnref:findface" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:orientation">
<p>Here's an
<a href="http://mashable.com/2017/09/11/artificial-intelligence-ai-lgbtq-gay-straight/">article</a>
that does a good job at explaining why this is terrible science (and ethics). <a class="footnote-backref" href="#fnref:orientation" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:gender">
<p>Example <a href="https://www.theverge.com/2017/8/22/16180080/transgender-youtubers-ai-facial-recognition-dataset">press
coverage</a>.
I think I've seen good criticism of it at the time but I can't find it
anymore =( <a class="footnote-backref" href="#fnref:gender" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:faceid">
<p>From Apple's <a href="https://images.apple.com/business/docs/FaceID_Security_Guide.pdf">FaceID Security
Guide</a>
(PDF). <a class="footnote-backref" href="#fnref:faceid" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
</ol>
</div>k-anonymity, the parent of all privacy definitions2017-08-14T00:00:00+02:002017-10-01T00:00:00+02:00Damien Desfontainestag:desfontain.es,2017-08-14:/privacy/k-anonymity.html<p>How a privacy researcher proved a politician wrong, and how she created the first ever definition of anonymity in the process.</p><p><strong>In 1997</strong>, a PhD student named <a href="https://en.wikipedia.org/wiki/Latanya_Sweeney">Latanya Sweeney</a> heard about an
interesting data release. A <a href="http://www.mass.gov/anf/employee-insurance-and-retirement-benefits/oversight-agencies/gic/">health insurance organization</a> from
Massachusetts had compiled a database of hospital visits by state employees, and
had thought that giving it to researchers could encourage innovation and
scientific discovery. Of course, there were privacy considerations: allowing
researchers to look at other citizens health records seemed pretty creepy. So
they decided to do the obvious thing, and remove all columns that indicated who
a patient was: name, phone number, full address, social security number, etc.</p>
<p>As you can probably guess, this didn't end so well. In this article, I'll
describe and analyze Sweeney's successful reidentification attack, and I'll
explain the privacy definition that Sweeney invented to prevent this type of
attack in the future: <span class="math">\(k\)</span>-anonymity.</p>
<div class="toc">
<ul>
<li><a href="#what-went-wrong">What went wrong?</a></li>
<li><a href="#how-to-prevent-this-attack">How to prevent this attack?</a><ul>
<li><a href="#definition-of-k-anonymity">Definition of \(k\)-anonymity</a><ul>
<li><a href="#what-types-of-data-are-reidentifying">What types of data are reidentifying?</a></li>
<li><a href="#how-to-choose-k">How to choose \(k\)?</a></li>
</ul>
</li>
<li><a href="#how-to-make-a-dataset-k-anonymous">How to make a dataset \(k\)-anonymous?</a><ul>
<li><a href="#building-block-1-generalization">Building block 1: generalization</a></li>
<li><a href="#two-types-of-generalization">Two types of generalization</a></li>
<li><a href="#building-block-2-suppression">Building block 2: suppression</a></li>
<li><a href="#algorithms">Algorithms</a></li>
<li><a href="#in-practice">In practice</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#how-convincing-really-is-k-anonymity">How convincing really is \(k\)-anonymity?</a></li>
</ul>
</div>
<p></p>
<h1 id="what-went-wrong">What went wrong?</h1>
<p>Some demographic information was left in the database, so researchers could
still compile useful stats: ZIP code, date of birth, and gender were all part
of the data. Sweeney realized that the claims of the Massachusetts governor,
who insisted that the privacy of state employees was respected (all identifiers
were removed!), were perhaps a little bit over-optimistic. Since the governor
himself was a state employee, Sweeney decided to do the obvious thing and
reidentify which records of the "anonymized" database were the governor's.</p>
<p>With just $20, Sweeney bought the public voter records from Massachusetts, which
had both full identifiers (names, addresses) and demographic data (ZIP code and
date of birth), and contained the governor's information. Guess how many records
matched the governor's gender, ZIP code, and date of birth inside the hospital
database? Only one, and thus, Sweeney was able to know which prescriptions and
visits in the data were the governor's. She posted all of it to his office,
showing theatrically that their anonymization process wasn't as solid as it
should have been.</p>
<p>Several factors made this attack possible. Some are obvious, but not all:</p>
<ol>
<li>
<p>The hospital data contained demographic information that could be used to
distinguish between different records.</p>
</li>
<li>
<p>A secondary database was available to figure out the demographic information
about the target.</p>
</li>
<li>
<p>The target was in both datasets.</p>
</li>
<li>
<p>And the demographic information of the target (ZIP code, date of birth, and
gender) was unique within both datasets: only one record had the demographic
values of the governor.</p>
</li>
</ol>
<p>At first glance, these factors appear to be <em>necessary</em>: remove one of them and
suddenly, the attack no longer works. (Try it! It's a good mental exercise.)</p>
<h1 id="how-to-prevent-this-attack">How to prevent this attack?</h1>
<p>As per our previous remark, removing one of the factors should be enough to
prevent attacks like these. Which ones can we afford to remove, while making
sure that the data can be used for data analysis tasks?</p>
<ol>
<li>
<p>We could remove all demographic information from the data, or even all
information that might be linked to a person using auxiliary sources.
Unfortunately, this would also severely hinder the utility of the data:
correlations based on age, gender, and geographic info are very useful to
researchers!</p>
</li>
<li>
<p>Society probably <em>should</em> do something about the existence of public (or
commercially available) data sources that can be used in reidentification
attacks. However, this is a complex political issue, so a little bit out of
scope for a data owner who wants to publish or share an anonymized version of
their data — in practice, there's pretty much nothing we can do about it.</p>
</li>
<li>
<p>Again, there's not much we can do. We have no way to modify the secondary
(public) dataset. We could decrease the probability that a random target is
in our dataset by sub-sampling it, but all people in the sample would still
be at risk, so this is obviously not a satisfying solution.</p>
</li>
<li>
<p>Now, this is the interesting point. Maybe suppressing all demographic values
would render the data useless, but there might be a middle ground to make
sure that the demographic values are no longer unique in the dataset.</p>
</li>
</ol>
<p>This last suggestion is the basic idea of <span class="math">\(k\)</span>-anonymity. </p>
<h2 id="definition-of-k-anonymity">Definition of <span class="math">\(k\)</span>-anonymity</h2>
<p>A dataset is said to <em>be <span class="math">\(k\)</span>-anonymous</em> if every combination of values for
demographic columns in the dataset appears at least for <em>k</em> different records.</p>
<p>For example, this dataset is <span class="math">\(2\)</span>-anonymous:</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">34</td>
</tr>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">34</td>
</tr>
<tr>
<td style="text-align: center;">1742</td>
<td style="text-align: center;">77</td>
</tr>
<tr>
<td style="text-align: center;">1742</td>
<td style="text-align: center;">77</td>
</tr>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">34</td>
</tr>
</tbody>
</table>
<p>This one isn't:</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">34</td>
</tr>
<tr>
<td style="text-align: center;">1742</td>
<td style="text-align: center;">77</td>
</tr>
<tr>
<td style="text-align: center;">1743</td>
<td style="text-align: center;">77</td>
</tr>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">34</td>
</tr>
</tbody>
</table>
<p>Notice that we need every <em>combination</em> of values to appear at least <span class="math">\(k\)</span> times.
Thus, even if each individual value of each column appears <span class="math">\(2\)</span> times in the
following dataset, it's not <span class="math">\(2\)</span>-anonymous:</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">34</td>
</tr>
<tr>
<td style="text-align: center;">1742</td>
<td style="text-align: center;">34</td>
</tr>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">77</td>
</tr>
<tr>
<td style="text-align: center;">1742</td>
<td style="text-align: center;">77</td>
</tr>
</tbody>
</table>
<p>The intuition is that when a dataset is <span class="math">\(k\)</span>-anonymous for a sufficiently large
<span class="math">\(k\)</span>, the last requirement for a successful reidentification attack is broken. An
attacker might find out the demographic information of their target using a
secondary database, but then this demographic information will be linked to <span class="math">\(k\)</span>
different individuals, so it will be impossible to know which one is their info.</p>
<h4 id="what-types-of-data-are-reidentifying">What types of data are reidentifying?</h4>
<p>Note that we've only talked about "demographic information", which is pretty
vague. ZIP codes, age, gender are all good candidates for reidentification
attacks, because they're public (or easily findable) information that is also
often found in sensitive datasets (especially medical ones). In general, the
data owner should consider which columns might be used by the attacker they're
concerned about.</p>
<p>These columns, not necessarily sensitive themselves but which might be used in a
reidentification attack, are called <em>quasi-identifiers</em> (or <em>QIs</em>). There is no
universal list of quasi-identifiers, it depends on the attack model. If some
data types are almost always QIs (ZIP code, age, gender…), many more depend on
the context (like timestamps, medical conditions, physical characteristics…).
The question to ask is: would the person who's trying to attack our dataset have
access to these values through public or commercially available data?</p>
<p><small> I'll try to write more about attack modeling and data classification
later. This is not as easily explainable as the various mathematical definitions
of privacy: it has lots of human components and as such, is always a bit fuzzy.
Which makes it even more interesting! :D But I digress. </small></p>
<h4 id="how-to-choose-k">How to choose <span class="math">\(k\)</span>?</h4>
<p>Short answer: ¯\_(ツ)_/¯</p>
<p>Longer answer: nobody knows. In the healthcare world, when medical data is
shared with a small number of people (typically for research purposes), <span class="math">\(k\)</span> is
often chosen between <span class="math">\(5\)</span> and <span class="math">\(15\)</span>. This choice is very arbitrary and ad hoc. To
the best of my knowledge, there is no official law or regulation which suggests
a specific value. Some universities, companies or other organizations have
official guidelines, but the vast majority don't.</p>
<p>To pick a parameter for a privacy definition, one needs to understand what's the
link between the parameter value, and the risk of a privacy incident happening.
But this is difficult: if <span class="math">\(k\)</span>-anonymity is relatively easy to understand,
estimating risk quantitatively is extremely tricky. <small> I'm also going to
write a bit about this later on! </small></p>
<ul>
<li>Regulators don't want to include specific parameter values in laws or
guidelines, since there is no convincing argument to be made for a given
choice, and the level of risk depends on many more fuzzy parameters (how
valuable the data is, how bad would a privacy incident be, etc.).</li>
<li>Data owners don't know how to choose the parameter either, so they usually buy
the services of a privacy consultant to do this choice (and take care of the
anonymization process). This consultant doesn't know either what's the "good"
choice, but they usually have more practical experience of what are common
values in the industry for similar levels of risk.</li>
</ul>
<p><small> This is my first "real" blog post, about the most basic anonymity
definition there is, and I've already reached my second digression to say
"notice how it's actually super fuzzy and thus, complicated to apply in
practice?". Isn't privacy fun? :D </small></p>
<h2 id="how-to-make-a-dataset-k-anonymous">How to make a dataset <span class="math">\(k\)</span>-anonymous?</h2>
<p>So, suppose we picked our quasi-identifiers and <span class="math">\(k=2\)</span>. Even with such a low
value for <span class="math">\(k\)</span>, our original dataset will likely not be <span class="math">\(k\)</span>-anonymous: there will
be many records with unique combinations of quasi-identifier values.</p>
<p>The two main building blocks used to transform a dataset into a <span class="math">\(k\)</span>-anonymous
table are <em>generalization</em> and <em>suppression</em>.</p>
<h4 id="building-block-1-generalization">Building block 1: generalization</h4>
<p>Generalization is the process of making a quasi-identifier value less precise,
so that records with different values are transformed (or <em>generalized</em>) into
records that share the same values. Consider the records in this table:</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">34</td>
</tr>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">39</td>
</tr>
<tr>
<td style="text-align: center;">1742</td>
<td style="text-align: center;">75</td>
</tr>
<tr>
<td style="text-align: center;">1691</td>
<td style="text-align: center;">77</td>
</tr>
</tbody>
</table>
<p>The numerical values of these records can be transformed into <em>numerical
ranges</em>, so that the resulting table verifies <span class="math">\(2\)</span>-anonymity:</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">30-39</td>
</tr>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">30-39</td>
</tr>
<tr>
<td style="text-align: center;">1***</td>
<td style="text-align: center;">75-79</td>
</tr>
<tr>
<td style="text-align: center;">1***</td>
<td style="text-align: center;">75-79</td>
</tr>
</tbody>
</table>
<p>The idea of generalization is to make demographic information more imprecise to
satisfy our privacy requirements, but still allow useful data analysis to be
done. In our example, changing precise ages into age ranges is probably enough
to analyze whether a disease affects young or old people disproportionately.</p>
<p>Transforming a numerical value into a range is one of the most typical ways of
performing generalization. Other ways include removing a value entirely (e.g.
transforming a gender value into "gender unknown"), or using a <em>generalization
hierarchy</em> (e.g. transforming an <a href="https://en.wikipedia.org/wiki/ICD-10">ICD-10 diagnosis code</a> into a
truncated code, or the corresponding <a href="https://en.wikipedia.org/wiki/ICD-10_Chapter_I:_Certain_infectious_and_parasitic_diseases">block</a>).</p>
<h4 id="two-types-of-generalization">Two types of generalization</h4>
<p>Generalization strategies can be classified into two categories: <em>global</em> and
<em>local</em>. Consider the following table:</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">34</td>
</tr>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">34</td>
</tr>
<tr>
<td style="text-align: center;">1742</td>
<td style="text-align: center;">34</td>
</tr>
<tr>
<td style="text-align: center;">1742</td>
<td style="text-align: center;">31</td>
</tr>
</tbody>
</table>
<p>Global generalization means that a given value for a given column will <em>always</em>
be generalized in the same way: if you decide to transform age 34 into age range
30-34 for one record, all records that have ages between 30 and 34 will be
transformed into this fixed range of 30-34. Using global generalization, the
example could be transformed into:</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">30-34</td>
</tr>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">30-34</td>
</tr>
<tr>
<td style="text-align: center;">1742</td>
<td style="text-align: center;">30-34</td>
</tr>
<tr>
<td style="text-align: center;">1742</td>
<td style="text-align: center;">30-34</td>
</tr>
</tbody>
</table>
<p>Local generalization doesn't have that constraint: it allows you to pick a
different generalization for each record. A value 34 in the age column might
stay untouched for one record, and generalized for other: </p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">34</td>
</tr>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">34</td>
</tr>
<tr>
<td style="text-align: center;">1742</td>
<td style="text-align: center;">30-34</td>
</tr>
<tr>
<td style="text-align: center;">1742</td>
<td style="text-align: center;">30-34</td>
</tr>
</tbody>
</table>
<p>Global generalization usually makes it easier to do data analysis on generalized
data; while local generalization allows to keep more utility at the cost of a
slightly more complex data representation.</p>
<h4 id="building-block-2-suppression">Building block 2: suppression</h4>
<p>In our previous example, our records had relatively "close" demographic values,
which allowed generalization to keep reasonably accurate information while still
ensuring <span class="math">\(2\)</span>-anonymity. What if the table is instead:</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">34</td>
</tr>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">39</td>
</tr>
<tr>
<td style="text-align: center;">1742</td>
<td style="text-align: center;">75</td>
</tr>
<tr>
<td style="text-align: center;">1691</td>
<td style="text-align: center;">77</td>
</tr>
<tr>
<td style="text-align: center;">9755</td>
<td style="text-align: center;">13</td>
</tr>
</tbody>
</table>
<p>The first four records can be grouped in two pairs as above, but the last record
is an outlier. Grouping it with one of the pairs above would mean having very
large ranges of values (age between 10 and 39, or ZIP code being completely
removed), which would significantly reduce the utility of the resulting data. So
a simple solution to deal with such outlier values is simply to remove them from
the data. Using both generalization and suppression on this example could lead
to the same <span class="math">\(2\)</span>-anonymous table as before:</p>
<table>
<thead>
<tr>
<th style="text-align: center;">ZIP code</th>
<th style="text-align: center;">age</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">30-39</td>
</tr>
<tr>
<td style="text-align: center;">4217</td>
<td style="text-align: center;">30-39</td>
</tr>
<tr>
<td style="text-align: center;">1000-1999</td>
<td style="text-align: center;">75-79</td>
</tr>
<tr>
<td style="text-align: center;">1000-1999</td>
<td style="text-align: center;">75-79</td>
</tr>
</tbody>
</table>
<p>Using this method, there are usually strictly less records in the transformed
table than in the original. On large datasets, allowing a small percentage of
suppressed records typically allows the result to be <span class="math">\(k\)</span>-anonymous without
requiring too much generalization.</p>
<h4 id="algorithms">Algorithms</h4>
<p><span class="math">\(k\)</span>-anonymity is the oldest privacy definition, it's relatively simple to
understand, so it has been quickly adopted by the healthcare community for their
data anonymization needs. As a result, there has been a <em>lot</em> of research on how
to transform a dataset into a <span class="math">\(k\)</span>-anonymous table.</p>
<p>The problem of finding an <em>optimal</em> strategy for <span class="math">\(k\)</span>-anonymity is <a href="https://en.wikipedia.org/wiki/NP-hardness">NP-hard</a>, for
basically any reasonable definition of optimality. <a href="https://desfontain.es/PDFs/PhD/OnTheComplexityOfOptimalKAnonymity.pdf">This paper<sup>
(pdf)</sup></a> presents a few such results, if you're interested in this
kind of thing ^^</p>
<p>A list of approximation algorithms for the optimal <span class="math">\(k\)</span>-anonymization problem can
be found in <a href="https://desfontain.es/PDFs/PhD/PublishingDataFromElectronicHealthRecordsWhilePreservingPrivacyASurveyOfAlgorithms.pdf">this paper<sup> (pdf)</sup></a> (Table 4, page 11). 18
different algorithms are listed, and I don't even think the list is exhaustive!
The paper contains many links to the original papers, and to some comparisons
between methods. Sadly, there is no unified benchmark to know how all these
algorithms perform on various data analysis tasks.</p>
<h4 id="in-practice">In practice</h4>
<p>Unless you're a PhD student working on your literature review, you're probably
not looking for a bunch of links to research papers about complicated
<span class="math">\(k\)</span>-anonymization algorithms. If you're a data owner trying to transform a
dataset to get a <span class="math">\(k\)</span>-anonymous table, you may be looking for software instead.</p>
<p>As of 2017, the main open-source tool for data anonymization is <a href="http://arx.deidentifier.org/">ARX</a>. Its
interface is a bit difficult to understand at first, but it works fairly well on
small to moderately large datasets, and implements a lot more than just
<span class="math">\(k\)</span>-anonymity algorithms. It used to feature only global generalization
techniques<sup id="fnref:edit"><a class="footnote-ref" href="#fn:edit">1</a></sup>, but this apparently <a href="http://arx.deidentifier.org/anonymization-tool/analysis/#a50">changed recently</a>.</p>
<p>There are other tools available online, but none of them is anywhere as usable
as ARX. Many of them are listed in the <a href="http://arx.deidentifier.org/overview/related-software/">Related software</a> page of ARX's
website. I've tried most of them, only to get convinced that none of them really
reached the point of being a usable product. <a href="http://cs.utdallas.edu/dspl/cgi-bin/toolbox/index.php">UTD Anonymization Toolbox</a> is
probably the only one worth a look: it requires to use the command-line and
impractical configuration files to work, but it implements a local
generalization algorithm (the first of its kind, named <a href="https://desfontain.es/PDFs/PhD/MondrianMultidimensionalKANonymity.pdf">Mondrian<sup>
(pdf)</sup></a>, a very cool technique with better utility preservation
than global generalization algorithms).</p>
<p>On the commercial side, I've only heard of a toolkit developed by the consulting
company <a href="https://privacy-analytics.com/">Privacy Analytics</a>. The intended audience seems to be people who know
little about privacy: it looks very shiny, but I didn't manage to understand
which anonymity property or algorithms they were using ^^ You can get a free
trial by filling up a form on their website, but I can only assume the real
version is very expensive, since there is no mention of price anywhere.</p>
<h1 id="how-convincing-really-is-k-anonymity">How convincing really is <span class="math">\(k\)</span>-anonymity?</h1>
<p><span class="math">\(k\)</span>-anonymity is simple to understand, and it seems intuitively obvious that
reidentification attacks are well mitigated when a dataset is transformed to
become <span class="math">\(k\)</span>-anonymous. However, it only mitigates this particular kind of attack.
We assumed that all that the attacker wanted was to select a target, point at a
record, and say "this record corresponds to my target" with a high certainty.
This matches Sweeney's original attack, but how realistic is this?</p>
<p>When an attacker successfully reidentifies someone in a dataset, it's not
necessarily a privacy issue. Consider the voter files from earlier. By law, this
data is public, and contains full names. It's very easy for an attacker to
point at a random record and shouting "hey, I reidentified this person!": the
identification is <em>right there</em> in the dataset. This "attack" <em>always succeeds</em>,
but it's not really interesting, nor particularly creepy… Why is that?</p>
<p>In Sweeney's example, the creepy thing isn't just finding the data subject
associated with a given record. The <em>sensitive</em> information linked with the
record (in our leading example, diagnostics and drug prescriptions) is where the
creepiness comes from! The leak of <em>sensitive</em> information associated to one
given individual is the problem, not the reidentification itself.</p>
<p><span class="math">\(k\)</span>-anonymity doesn't really capture this idea. The definition just prevents you
from knowing the real identity of an anonymized record. But maybe there are
other attacks that allow you to find out sensitive information about someone,
without finding with absolute certainty which record is theirs?</p>
<p>As I'll explain in future articles, other types of attacks do exist, and many
other definitions have been proposed in order to mitigate them too. Nonetheless,
<span class="math">\(k\)</span>-anonymity is still used in the healthcare world, in large part because of
its simplicity and utility preservation compared to other definitions.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:edit">
<p>A previous version of this post claimed that only global generalization
was available in ARX. Sorry for the factual mistake! I should have read the
docs more closely =) <a class="footnote-backref" href="#fnref:edit" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Beginnings2017-07-17T00:00:00+02:002017-07-17T00:00:00+02:00Damien Desfontainestag:desfontain.es,2017-07-17:/privacy/beginnings.html<p>Blog intro. What's going to be there?</p><p><strong>Hi there!</strong> I'm <a href="../serious.html">Damien</a>. I have no idea how people usually
start blogs, so bear with me while I figure this out.</p>
<p>I see this place as a way to publish things that are too long for Twitter, too
opinionated for Wikipedia, and not pretentious enough for Medium. I'm going to
try and keep it to three themes: <em>privacy</em>, <em>research</em>, and <em>privacy research</em>.
I'm not (yet) a specialist in any of these. Hopefully, thanks for my jobs and
personal interest in those topics, I can add something valuable to what's
written online about them.</p>
<p>The following is the vision I have of these three themes. This should give an
idea of what I intend to talk about in this blog =)</p>
<h2 id="privacy">Privacy</h2>
<p>It's difficult to define what privacy encompasses. It's easier to realize when
you don't have enough privacy — through bad surprises, uneasy feelings of
creepiness, or real risks to your safety.</p>
<p>When a parent or a partner installs stealthy software on your phone to spy on
your texts and calls, that's an invasion of your privacy. When a company sells
your name, address and purchase history to some sketchy third-party that sends
you targeted ads, the uneasy feeling you get comes from a lack of privacy.
Full-body scanners in certain airports are an attack to one's bodily privacy.
Data leaks are a risk to users' privacy.</p>
<p>Privacy issues usually come from a lack of <em>transparency</em>, of <em>control</em>, or
both. In an ideal world, everybody would know exactly who has access to which
data about them and why. Personal data collection would not happen without
informed consent, and people would have a right to access, modify and delete
data that other people or organizations hold about them.</p>
<p>The fuzziness, and the complexity of the issues in this space, are part of what
I find interesting about them. I have done many privacy reviews for Google
products, and there is always something interesting and new with each of them.
Would users expect this behavior? Is this deletion action clear enough? Could
someone re-identify this aggregated data?</p>
<p>Like security, privacy is of particular importance for marginalized communities.
Having your phone number leaked online is much more problematic if you're a
high-profile political activist, or a closeted LGBTQ+ blogger. Harassment of
folks that belong to minorities is a major problem, and badly-designed sharing
interfaces or insufficient anti-abuse tools can lead to dramatic consequences.
Designing tools that deal with potentially sensitive data, and failing to
consider these specific risks, is highly irresponsible. And you can easily guess
what I think of compliance-based privacy programs…</p>
<p>I also try to avoid absolutist viewpoints. They are hardly ever constructive,
and they are often dangerous. I know people who refuse to use Signal because
it's not available without Google Play Services, while continuing to communicate
via cleartext SMS messages. For most practical problems, there is no perfect
solution. Focusing on defending against a hypothetical all-powerful targeted
attacker is usually pointless. Instead, I try to focus on realistic threat
models, usable tools, and risk mitigation.</p>
<h2 id="research">Research</h2>
<p>I started a part-time PhD after two years of software engineering at Google.</p>
<p>To solve an engineering problem, the path is quite straightforward. Grasp the
scope of the problem, design a solution, validate the design with coworkers and
stakeholders, write code, verify that the solution is "good enough", then
productionize it. Once the problem disappeared, there's no time to think about
it more: there are other problems to solve, other fires to put out.</p>
<p>The whole process is fun and rewarding, but I'm frustrated by the ending. What
if we could design a simpler or more efficient solution? Prove that it works in
a wider range of situations? Share the idea behind it with more people, and see
whether they get inspired and solve other problems? Doing all of this is not
immediately rewarding, but I think it can have a deeper, and longer-lasting
impact, than core engineering work.</p>
<p>I optimistically think that academia is the place to do that. Compare the
solution to what's out there already, make more experiments, write proofs,
figure out what additional impact it could have. Share the results with as many
people as possible. It might not be worth the time, but I think it's worthwhile
to give it a try. There are certainly interesting things to learn along the way.</p>
<p>The one thing that I'm afraid of is spending time solving the wrong problems.
Finding a "good problem" is not easy: a good problem must be difficult enough to
not have been already solved, but simple enough to have a chance at tackling it.
Identifying practical problems and their precise constraints is also hard, when
the main source of inspiration is other academics' work.</p>
<p>I'm frustrated about the lack of incentives to do research work as a software
engineer, but the incentives of academia are even more broken. Publication
metrics are a bad way to estimate one's impact, especially in the short term.
The peer review process is terribly implemented in practice. The whole system
makes it painfully slow to gather feedback, and the little feedback you get is
imprecise. The idea of having my work praised only to realize much later that it
didn't make a difference in practice… It's even scarier to me than the idea of
not finding joy and impact in my research, and deciding to quit.</p>
<p>But I'm not exactly pessimistic :D I feel lucky and enthusiastic about this
part-time project. Continuing to do engineering work for Google gives me an
endless input of complicated real-world problems to tackle, many of which seem
to be good candidates for research projects. I am surrounded by impressively
smart and passionate coworkers on both sides, whose feedback is invaluable. And
I don't feel extremely attached to the idea of having an academic career or even
getting the title at the end of my PhD, so I don't really feel the pressure to
publish everything and anything just to increment some counters.</p>
<p>All in all, this sounds like a fun and challenging adventure. I'm excited to see
what I'll learn along the way!</p>
<h2 id="privacy-research">Privacy research</h2>
<p>My research, like my engineering job at Google, will focus on privacy. This is a
field whose boundaries are not very well-defined, and that has very distinct
sub-fields. Some researchers focus on user research to understand the
perceptions of real people with regard to their personal data (there are a bunch
of them at Google). Very little math is involved. Some are designing algorithms
that have provable privacy-related properties, like private set intersection or
differentially private surveys. Lots of math there! ^^ Some study the problem of
<em>anonymizing</em> (or <em>de-identifying</em>) a dataset, so it can be used by more people
or shared with third parties. Some focus on onion routing, on online tracking,
on cryptocurrency, on privacy policies, on genetic privacy, on social networks,
and the list is far from exhaustive. So… what am I doing exactly?</p>
<p>My PhD project is about <em>making it easier for data owners to understand and
protect the personal information contained in their databases</em>. I see this goal
as having two main subcomponents.</p>
<ol>
<li>
<p><em>Risk analysis</em>. There are lots of organizations, companies or governments
which sit on large databases with personal information, and it's difficult
for them to realize how sensitive it is. Leaking your users' country of
origin is intuitively less of a problem than leaking their e-mail addresses,
which in turn is not as big a deal as leaking their credit card information.
Sadly, doing this type of inventory and risk analysis is currently pretty
difficult: it requires time, investment, and specific expertise. It
shouldn't have to be this way, so I'm working towards building tools that
make this easier.</p>
</li>
<li>
<p><em>Anonymization</em>. Once you realized how sensitive your data is, you hopefully
will want to take steps to protect it. There are many ways to lower the risk
of bad people having access to your database: encryption, access controls,
or many other security techniques. Another option is to modify the database,
in a way that makes sure that somebody with access to it can't deduce creepy
things about the individuals whose data is in the database. I'm working
towards making this process easier and more understandable for data owners.</p>
</li>
</ol>
<p>I could (and hopefully, I will!) talk at length about these two things. They
have already been studied by many people over the past ~15 years (especially
anonymization), but I think that there is a lot of room for more vulgarization
on the topic, and significant improvements to do on the research side. On the
anonymization topic in particular, I feel like it is urgent to work towards
bridging the gap between research advances and concrete use cases. </p>
<p>Maybe I'll realize along the way that I'm looking at the wrong problems, or that
it proves more difficult than I thought to improve the state of the art. But as
I've been told, that's part of what makes it challenging and fun ^^</p>