--- /dev/null
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
+ "http://www.w3.org/TR/html4/loose.dtd">
+ <html>
+ <head><title>LWN.net Weekly Edition for August 30, 2018 [LWN.net]</title>
+ <link rel="next" href="/Articles/763254/"/>
+<meta name="twitter:card" content="summary" />
+<meta name="twitter:site" content="@lwnnet" />
+<meta name="twitter:title" content="LWN.net Weekly Edition for August 30, 2018" />
+<meta name="twitter:description" content="Julia; C considered dangerous; 4.19 Merge window; I/O controller throughput; KDE onboarding; Dat." />
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8">
+ <link rel="icon" href="/images/favicon.png" type="image/png">
+ <link rel="alternate" type="application/rss+xml" title="LWN.net headlines" href="https://lwn.net/headlines/newrss">
+ <link rel="stylesheet" href="/CSS/lwn">
+<link rel="stylesheet" href="/CSS/nosub">
+<link rel="stylesheet" href="/CSS/pure-min">
+ <!--[if lte IE 8]>
+ <link rel="stylesheet" href="/CSS/grids-responsive-old-ie-min">
+ <![endif]-->
+ <!--[if gt IE 8]><!-->
+ <link rel="stylesheet" href="/CSS/grids-responsive-min">
+ <!--<![endif]-->
+ <link rel="stylesheet" href="/CSS/pure-lwn">
+
+
+<script type="text/javascript">var p="http",d="static";if(document.location.protocol=="https:"){p+="s";d="engine";}var z=document.createElement("script");z.type="text/javascript";z.async=true;z.src=p+"://"+d+".adzerk.net/ados.js";var s=document.getElementsByTagName("script")[0];s.parentNode.insertBefore(z,s);</script>
+<script type="text/javascript">
+var ados_keywords = ados_keywords || [];
+if( location.protocol=='https:' ) {
+ ados_keywords.push('T:SSL');
+} else {
+ ados_keywords.push('T:HTTP');
+}
+
+var ados = ados || {};
+ados.run = ados.run || [];
+ados.run.push(function() {
+
+ados_add_placement(4669, 20979, "azk13321_leaderboard", 4).setZone(16026);
+
+ados_add_placement(4669, 20979, "azk93271_right_zone", [5,10,6]).setZone(16027);
+
+ados_add_placement(4669, 20979, "azk31017_tracking", 20).setZone(20995);
+
+
+
+ados_setKeywords(ados_keywords.join(', '));
+ados_load();
+});</script>
+
+ </head>
+ <body bgcolor="#ffffff" link="Blue" VLINK="Green" alink="Green">
+ <a name="t"></a>
+<div id="menu"><a href="/"><img src="https://static.lwn.net/images/logo/barepenguin-70.png" class="logo"
+ border="0" alt="LWN.net Logo">
+ <font class="logo">LWN<br>.net</font>
+ <font class="logobl">News from the source</font></a>
+ <a href="/"><img src="https://static.lwn.net/images/lcorner-ss.png" class="sslogo"
+ border="0" alt="LWN"></a><div class="navmenu-container">
+ <ul class="navmenu">
+ <li><a class="navmenu" href="#t"><b>Content</b></a><ul><li><a href="/current/">Weekly Edition</a></li><li><a href="/Archives/">Archives</a></li><li><a href="/Search/">Search</a></li><li><a href="/Kernel/">Kernel</a></li><li><a href="/Security/">Security</a></li><li><a href="/Distributions/">Distributions</a></li><li><a href="/Calendar/">Events calendar</a></li><li><a href="/Comments/unread">Unread comments</a></li><li><hr></li><li><a href="/op/FAQ.lwn">LWN FAQ</a></li><li><a href="/op/AuthorGuide.lwn">Write for us</a></li></ul></li>
+<li><a class="navmenu" href="#t"><b>Edition</b></a><ul><li><a href="/Articles/763252/">⇒Front page</a></li><li><a href="/Articles/763254/">Brief items</a></li><li><a href="/Articles/763255/">Announcements</a></li><li><a href="/Articles/763252/bigpage">One big page</a></li><li><a href="/Articles/762816/">Previous week</a></li><li><a href="/Articles/763789/">Following week</a></li></ul></li>
+</ul></div>
+</div> <!-- menu -->
+<div class="pure-g not-handset" style="margin-left: 10.5em">
+ <div class="not-print">
+ <div id="azk13321_leaderboard"></div>
+ </div>
+ </div>
+ <div class="topnav-container">
+<div class="not-handset"><form action="https://lwn.net/Login/" method="post" name="loginform"
+ class="loginform">
+ <b>User:</b> <input type="text" name="Username" value="" size="8" /> <b>Password:</b> <input type="password" name="Password" size="8" /> <input type="hidden" name="target" value="/Articles/763252/" /> <input type="submit" name="submit" value="Log in" /></form> |
+ <form action="https://lwn.net/subscribe/" method="post" class="loginform">
+ <input type="submit" name="submit" value="Subscribe" />
+ </form> |
+ <form action="https://lwn.net/Login/newaccount" method="post" class="loginform">
+ <input type="submit" name="submit" value="Register" />
+ </form>
+ </div>
+ <div class="handset-only">
+ <a href="/subscribe/"><b>Subscribe</b></a> /
+ <a href="/Login/"><b>Log in</b></a> /
+ <a href="/Login/newaccount"><b>New account</b></a>
+ </div>
+ </div><div class="pure-grid maincolumn">
+<div class="lwn-u-1 pure-u-md-19-24">
+<div class="PageHeadline">
+<h1>LWN.net Weekly Edition for August 30, 2018</h1>
+</div>
+<div class="ArticleText">
+<a name="763743"></a><h2 class="SummaryHL"><a href="/Articles/763743/">Welcome to the LWN.net Weekly Edition for August 30, 2018</a></h2>
+
+This edition contains the following feature content:
+ <p>
+ <ul class="spacylist">
+
+ <li> <a href="/Articles/763626/">An introduction to the Julia language,
+ part 1</a>: Julia is a language designed for intensive numerical
+ calculations; this article gives an overview of its core features.
+
+ <li> <a href="/Articles/763641/">C considered dangerous</a>: a Linux
+ Security Summit talk on what is being done to make the use of C in the
+ kernel safer.
+
+ <li> <a href="/Articles/763106/">The second half of the 4.19 merge
+ window</a>: the final features merged (or not merged) before the merge
+ window closed for this cycle.
+
+
+ <li> <a href="/Articles/763603/">Measuring (and fixing) I/O-controller
+ throughput loss</a>: the kernel's I/O controllers can provide useful
+ bandwidth guarantees, but at a significant cost in throughput.
+
+ <li> <a href="/Articles/763175/">KDE's onboarding initiative, one year
+ later</a>: what has gone right in KDE's effort to make it easier for
+ contributors to join the project, and what remains to be done.
+
+ <li> <a href="/Articles/763492/">Sharing and archiving data sets with
+ Dat</a>: an innovative approach to addressing and sharing data on the
+ net.
+
+</ul>
+
+<p>
+ This week's edition also includes these inner pages:
+ <p>
+ <ul class="spacylist">
+
+<li> <a href="/Articles/763254/">Brief items</a>: Brief news items from throughout the community.
+
+<li> <a href="/Articles/763255/">Announcements</a>: Newsletters, conferences, security updates, patches, and more.
+
+</ul>
+ <p>
+ Please enjoy this week's edition, and, as always, thank you for
+ supporting LWN.net.
+<p><a href="/Articles/763743/#Comments">Comments (none posted)</a>
+<p>
+<a name="763626"></a><h2 class="SummaryHL"><a href="/Articles/763626/">An introduction to the Julia language, part 1</a></h2>
+
+<div class="GAByline">
+ <p>August 28, 2018</p>
+ <p>This article was contributed by Lee Phillips</p>
+ </div>
+<p><a href="http://julialang.org/">Julia</a> is a young computer language
+aimed at serving the needs of scientists, engineers, and other
+practitioners of numerically intensive programming. It was first publicly
+released in 2012. After an intense period of language development, version
+1.0 was <a
+href="https://julialang.org/blog/2018/08/one-point-zero">released</a> on
+August 8. The 1.0 release promises years of language
+stability; users can be confident that developments in the 1.x series will
+not break their code.
+ This is the first part of a two-part article introducing the world of Julia.
+ This part will introduce enough of the language syntax and constructs to
+ allow you to begin to write simple programs. The following installment will
+ acquaint you with the additional pieces needed to create real projects, and to
+ make use of Julia's ecosystem.
+
+<h4>Goals and history</h4>
+
+<p>The Julia project has ambitious goals. It wants the language to perform
+about as well as Fortran or C when running numerical algorithms, while
+remaining as pleasant to program in as Python. I believe the project has
+met these goals and is poised to see increasing adoption by numerical
+researchers, especially now that an official, stable release is
+available.</p>
+
+<p>The Julia project maintains a <a
+href="https://julialang.org/benchmarks/">micro-benchmark page</a> that compares its
+numerical performance against both statically compiled languages (C,
+Fortran) and dynamically typed languages (R, Python). While it's certainly
+possible to argue about the relevance and fairness of particular
+benchmarks, the data overall supports the Julia team's contention that Julia
+has generally achieved parity with Fortran and C; the benchmark
+source code is available.</p>
+
+<p>Julia began as research in computer science at MIT; its creators are
+Alan Edelman, Stefan Karpinski, Jeff Bezanson, and Viral Shah. These four
+ remain active developers of the language. They, along with Keno Fischer,
+co-founder and CTO of <a href="https://juliacomputing.com/">Julia
+Computing</a>, were kind enough to share their thoughts with us about
+the language. I'll be drawing
+on their comments later on; for now, let's get a taste of
+what Julia code looks like.</p>
+
+<h4>Getting started</h4>
+
+<p>To explore Julia initially, start up its standard <a
+href="https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop">read-eval-print
+loop</a> (REPL)
+by typing <code>julia</code> at the terminal, assuming that you have installed
+it. You will then be
+able to interact with what will seem to be an interpreted language — but,
+behind the scenes, those commands are being compiled by a
+just-in-time (JIT) compiler that uses the <a href="http://llvm.org/">LLVM
+compiler framework</a>. This allows Julia to be interactive, while turning
+the code into fast, native machine instructions. However, the JIT compiler
+passes sometimes introduce noticeable delays at the REPL, especially when
+using a function for the first time.</p>
+
+<p>To run a Julia program non-interactively, execute a command like:
+<pre>
+ $ julia script.jl <args>
+</pre>
+
+<p>Julia has all the usual data structures: numbers of various types
+(including complex and rational numbers), multidimensional arrays,
+dictionaries, strings, and characters. Functions are first-class: they can
+be passed as arguments to other functions, can be members of arrays,
+and so on.</p>
+
+<p>Julia embraces Unicode. Strings, which are enclosed in double quotes,
+are arrays of Unicode characters, which are enclosed in single quotes. The
+"<tt>*</tt>" operator is used for string and character concatenation. Thus
+'a' and 'β' are characters, and 'aβ' is a syntax error. "a" and
+"β" are strings, as are "aβ", 'a' * 'β', and
+"a" * "β" — all evaluate to the same string.
+
+<p>Variable and function names can contain non-ASCII characters. This, along
+with Julia's clever syntax that understands numbers prepended to variables
+to mean multiplication, goes a long way to allowing the numerical scientist
+to write code that more closely resembles the compact mathematical notation
+of the equations that usually lie behind it.</p>
+
+<pre>
+ julia> ε₁ = 0.01
+ 0.01
+
+ julia> ε₂ = 0.02
+ 0.02
+
+ julia> 2ε₁ + 3ε₂
+ 0.08
+</pre>
+
+<p>And where does Julia come down on the age-old debate of what do about
+<tt>1/2</tt>? In Fortran and Python 2, this will get you 0, since 1 and 2 are
+integers, and the result is rounded down to the integer 0. This was deemed
+inconsistent, and confusing to some, so it was changed in Python 3 to
+return 0.5 — which is what you
+get in Julia, too.</p>
+
+<p>While we're on the subject of fractions, Julia can handle rational
+numbers, with a special syntax: <tt>3//5 + 2//3</tt> returns
+<tt>19//15</tt>, while <tt>3/5 + 2/3</tt>
+gets you the floating-point answer 1.2666666666666666. Internally, Julia
+thinks of a rational number in its reduced form, so the expression
+<tt>6//8 == 3//4</tt> returns <code>true</code>, and <code>numerator(6//8)</code> returns
+<code>3</code>.</p>
+
+<h4>Arrays</h4>
+
+<p>Arrays are enclosed in square brackets and indexed with an iterator that
+can contain a step value:</p>
+
+<pre>
+ julia> a = [1, 2, 3, 4, 5, 6]
+ 6-element Array{Int64,1}:
+ 1
+ 2
+ 3
+ 4
+ 5
+ 6
+
+ julia> a[1:2:end]
+ 3-element Array{Int64,1}:
+ 1
+ 3
+ 5
+</pre>
+
+<p>As you can see, indexing starts at one, and the useful <code>end</code>
+index means the obvious thing. When you define a variable in the REPL,
+Julia replies with the type and value of the assigned data; you can suppress this output by ending your input line with a semicolon.</p>
+
+<p>Since arrays are such a vital part of numerical computation, and Julia
+makes them easy to work with, we'll spend a bit more time with them than the other data structures.</p>
+
+<p>To illustrate the syntax, we can start with a couple of 2D arrays, defined at the REPL:</p>
+
+<pre>
+ julia> a = [1 2 3; 4 5 6]
+ 2×3 Array{Int64,2}:
+ 1 2 3
+ 4 5 6
+
+ julia> z = [-1 -2 -3; -4 -5 -6];
+</pre>
+
+<p>Indexing is as expected:</p>
+
+<pre>
+ julia> a[1, 2]
+ 2
+</pre>
+
+<p>You can glue arrays together horizontally:</p>
+
+<pre>
+ julia> [a z]
+ 2×6 Array{Int64,2}:
+ 1 2 3 -1 -2 -3
+ 4 5 6 -4 -5 -6
+</pre>
+
+<p>And vertically:</p>
+
+<pre>
+ julia> [a; z]
+ 4×3 Array{Int64,2}:
+ 1 2 3
+ 4 5 6
+ -1 -2 -3
+ -4 -5 -6
+</pre>
+
+<p>Julia has all the usual operators for handling arrays, and <a
+href="http://www.3blue1brown.com/essence-of-linear-algebra-page/">linear
+algebra</a> functions that work with matrices (2D arrays). The linear
+algebra functions are part of Julia's standard library, but need to be
+imported with a command like "<code>using LinearAlgebra</code>", which is a detail
+omitted from the current documentation. The functions include such things as
+determinants, matrix inverses, eigenvalues and eigenvectors, many kinds of
+matrix factorizations, etc. Julia has not reinvented the wheel here, but
+wisely uses the <a href="http://www.netlib.org/lapack/">LAPACK</a> Fortran
+library of battle-tested linear algebra routines.</p>
+
+<p>The extension of arithmetic operators to arrays is usually intuitive:</p>
+
+<pre>
+ julia> a + z
+ 2×3 Array{Int64,2}:
+ 0 0 0
+ 0 0 0
+</pre>
+
+<p>And the numerical prepending syntax works with arrays, too:</p>
+
+<pre>
+ julia> 3a + 4z
+ 2×3 Array{Int64,2}:
+ -1 -2 -3
+ -4 -5 -6
+</pre>
+
+<p>Putting a multiplication operator between two matrices gets you matrix
+multiplication:</p>
+
+<pre>
+ julia> a * transpose(a)
+ 2×2 Array{Int64,2}:
+ 14 32
+ 32 77
+</pre>
+
+<p>You can "broadcast" numbers to cover all the elements in an
+array by prepending the usual arithmetic operators with a dot:</p>
+
+<pre>
+ julia> 1 .+ a
+ 2×3 Array{Int64,2}:
+ 2 3 4
+ 5 6 7
+</pre>
+
+<p>Note that the language only actually requires the dot for some
+operators, but not for others, such as "*" and "/". The
+reasons for this are arcane, and it probably makes sense to be consistent
+and use the dot whenever you intend broadcasting. Note also that the
+current version of the official documentation is incorrect in claiming that
+you may omit the dot from "+" and "-"; in fact, this
+now gives an error.</p>
+
+<p>You can use the dot notation to turn any function into one that operates
+on each element of an array:</p>
+
+<pre>
+ julia> round.(sin.([0, π/2, π, 3π/2, 2π]))
+ 5-element Array{Float64,1}:
+ 0.0
+ 1.0
+ 0.0
+ -1.0
+ -0.0
+</pre>
+
+<p>The example above illustrates chaining two dotted functions
+together. The Julia compiler turns expressions like this into
+"fused" operations: instead of applying each function in turn to
+create a new array that is passed to the next function, the compiler
+combines the functions into a single compound function that is applied once
+over the array, creating a significant optimization.</p>
+
+<p>You can use this dot notation with any function, including your own, to
+turn it into a version that operates element-wise over arrays.</p>
+
+<p>Dictionaries (associative arrays) can be defined with several
+syntaxes. Here's one:</p>
+
+<pre>
+ julia> d1 = Dict("A"=>1, "B"=>2)
+ Dict{String,Int64} with 2 entries:
+ "B" => 2
+ "A" => 1
+</pre>
+
+<p>You may have noticed that the code snippets so far have not included any
+type declarations. Every value in Julia has a type, but the compiler will
+infer types if they are not specified. It is generally not necessary to
+declare types for performance, but type declarations sometimes serve other
+purposes, that we'll return to later. Julia has a deep and sophisticated
+type system, including user-defined types and C-like structs. Types can
+have behaviors associated with them, and can inherit behaviors from other
+types. The best thing about Julia's type system is that you can ignore it
+entirely, use just a few pieces of it, or spend weeks studying its
+design.</p>
+
+<h4>Control flow</h4>
+
+<p>Julia code is organized in blocks, which can indicate control flow,
+function definitions, and other code units. Blocks are terminated with the
+<code>end</code> keyword, and indentation is not significant. Statements
+are separated either with newlines or semicolons.</p>
+
+<p>Julia has the typical control flow constructs; here is a
+<code>while</code> block:</p>
+
+<pre>
+ julia> i = 1;
+
+ julia> while i < 5
+ print(i)
+ global i = i + 1
+ end
+ 1234
+</pre>
+
+<p>Notice the <code>global</code> keyword. Most blocks in Julia introduce a
+local scope for variables; without this keyword here, we would get an error
+about an undefined variable.</p>
+
+<p>Julia has the usual <code>if</code> statements and <code>for</code>
+loops that use the same iterators that we introduced above for array
+indexing. We can also iterate over collections:</p>
+
+<pre>
+ julia> for i ∈ ['a', 'b', 'c']
+ println(i)
+ end
+ a
+ b
+ c
+</pre>
+
+<p>In place of the fancy math symbol in this <code>for</code> loop, we can
+use "<tt>=</tt>" or "<tt>in</tt>". If you want to use
+the math symbol but
+have no convenient way to type it, the REPL will help you: type
+"<tt>\in</tt>" and the TAB key, and the symbol appears; you can type many
+<a href="/Articles/657157/">LaTeX</a> expressions into the
+REPL in this way.</p>
+
+<h4>Development of Julia</h4>
+
+<p>The language is developed on GitHub, with over 700 contributors. The
+Julia team mentioned in their email to us that the decision to use GitHub
+has been particularly good for Julia, as it streamlined the process for
+many of their contributors, who are scientists or domain experts in various
+fields, rather than professional software developers.</p>
+
+<p>The creators of Julia have <a
+href="https://julialang.org/publications/julia-fresh-approach-BEKS.pdf">published
+[PDF]</a>
+a detailed “mission statement” for the language, describing their aims and
+motivations. A key issue that they wanted their language to solve is what
+they called the "two-language problem." This situation is
+familiar to anyone who has used Python or another dynamic language on a
+demanding numerical problem. To get good performance, you will wind up
+rewriting the numerically intensive parts of the program in C or Fortran,
+dealing with the interface between the two languages, and may still be
+disappointed in the overhead presented by calling the foreign routines from
+your original code.
+
+<p>
+For Python, <a
+href="/Articles/738915/">NumPy and SciPy</a> wrap many
+numerical routines, written in Fortran or C, for efficient use from that
+language, but you can only take advantage of this if your calculation fits
+the pattern of an available routine; in more general cases, where you will
+have to write a loop over your data, you are stuck with Python's native
+performance, which is orders of magnitude slower. If you switch to an
+alternative, faster implementation of Python, such as <a
+href="https://pypy.org/">PyPy</a>, the numerical libraries may not be
+compatible; NumPy became available for PyPy only within about the past
+year.</p>
+
+<p>Julia solves the two-language problem by being as expressive and simple
+to program in as a dynamic scripting language, while having the native
+performance of a static, compiled language. There is no need to write
+numerical libraries in a second language, but C or Fortran library routines
+can be called using a facility that Julia has built-in. Other languages,
+such as <a href="https://github.com/JuliaPy/PyCall.jl">Python</a> or <a
+href="https://github.com/JuliaInterop/RCall.jl">R</a>, can also interoperate
+easily with Julia using external packages.</p>
+
+<h4>Documentation</h4>
+
+<p>There are many resources to turn to to learn the language. There is an
+extensive and detailed <a
+href="https://docs.julialang.org/en/stable/">manual</a> at Julia
+headquarters, and this may be a good place to start. However, although the
+first few chapters provide a gentle introduction, the material soon becomes
+dense and, at times, hard to follow, with references to concepts that are
+not explained until later chapters. Fortunately, there is a <a
+href="https://julialang.org/learning/">"learning" link</a> at the
+top of the Julia home page, which takes you to a long list of videos,
+tutorials, books, articles, and classes both about Julia and that use Julia
+in teaching subjects such a numerical analysis. There is also a fairly good
+ <a
+href="http://bogumilkaminski.pl/files/julia_express.pdf">cheat-sheet [PDF]</a>, which was
+just updated for v. 1.0.</p>
+
+<p>If you're coming from Python, <a
+href="https://docs.julialang.org/en/stable/manual/noteworthy-differences/#Noteworthy-differences-from-Python-1">this
+list</a> of noteworthy differences between Python and Julia syntax will
+probably be useful.</p>
+
+<p>Some of the linked tutorials are in the form of <a
+href="https://lwn.net/Articles/746386/">Jupyter notebooks</a> — indeed,
+the name "Jupyter" is formed from "Julia",
+"Python", and "R", which are the three original languages supported by
+the interface. The <a href="https://github.com/JuliaLang/IJulia.jl">Julia
+kernel for Jupyter</a> was recently upgraded to support v. 1.0. Judicious
+sampling of a variety of documentation sources, combined with liberal
+experimentation, may be the best way of learning the language. Jupyter
+makes this experimentation more inviting for those who enjoy the web-based
+interface, but the REPL that comes with Julia helps a great deal in this
+regard by providing, for instance, TAB completion and an extensive help
+system invoked by simply pressing the "?" key.</p>
+
+<h4>Stay tuned</h4>
+
+<p>
+ The <a href="/Articles/764001/">next installment</a> in this two-part series will explain how Julia is
+ organized around the concept of "multiple dispatch". You will learn how to
+ create functions and make elementary use of Julia's type system. We'll see how
+ to install packages and use modules, and how to make graphs. Finally, Part 2
+ will briefly survey the important topics of macros and distributed computing.
+<p><a href="/Articles/763626/#Comments">Comments (80 posted)</a>
+<p>
+<a name="763641"></a><h2 class="SummaryHL"><a href="/Articles/763641/">C considered dangerous</a></h2>
+
+<div class="FeatureByline">
+ By <b>Jake Edge</b><br>August 29, 2018
+ <hr>
+<a href="/Archives/ConferenceByYear/#2018-Linux_Security_Summit_NA">LSS NA</a>
+</div>
+<p>
+At the North America edition of the <a
+href="https://events.linuxfoundation.org/events/linux-security-summit-north-america-2018/">2018
+Linux Security Summit</a> (LSS NA), which was held in late August in Vancouver,
+Canada, Kees Cook gave a presentation on some of the dangers that come with
+programs written in C. In particular, of course, the Linux kernel is
+mostly written in C, which means that the security of our systems rests on
+a somewhat dangerous foundation. But there are things that can be done to
+help firm things up by "<span>Making C Less Dangerous</span>" as the title
+of his talk suggested.
+</p>
+
+<p>
+He began with a brief summary of the work that he and others are doing as
+part of the <a
+href="https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project">Kernel
+Self Protection Project</a> (KSPP). The goal of the project is to get
+kernel protections merged into the mainline. These protections are not
+targeted at protecting user-space processes from other (possibly rogue)
+processes, but are, instead, focused on protecting the kernel from
+user-space code. There are around 12 organizations and ten individuals
+working on roughly 20 different technologies as part of the KSPP, he said. The
+progress has been "slow and steady", he said, which is how he thinks it
+should go.
+</p>
+
+<a href="/Articles/763644/">
+<img src="https://static.lwn.net/images/2018/lssna-cook-sm.jpg" border=0 hspace=5 align="right"
+alt="[Kees Cook]" title="Kees Cook" width=214 height=300>
+</a>
+
+<p>
+One of the main problems is that C is treated mostly like a fancy assembler.
+The kernel developers do this because they want the kernel to be as fast
+and as small as possible. There are other reasons, too, such as the need to do
+architecture-specific tasks that lack a C API (e.g. setting up page tables,
+switching to 64-bit mode).
+</p>
+
+<p>
+But there is lots of undefined behavior in C. This "operational baggage"
+can lead to various problems. In addition, C has a weak standard library
+with multiple utility functions that have various pitfalls. In C, the content
+of uninitialized automatic variables is undefined, but in the machine code that it
+gets translated to, the value is whatever happened to be in that memory
+location before. In C, a function pointer can be called even if the type
+of the pointer does not match the type of the function being
+called—assembly doesn't care, it just jumps to a location, he said.
+</p>
+
+<p>
+The APIs in the standard library are also bad in many cases. He asked: why
+is there no argument to <tt>memcpy()</tt> to specify the maximum
+destination length? He noted a recent <a
+href="https://raphlinus.github.io/programming/rust/2018/08/17/undefined-behavior.html">blog
+post</a> from Raph Levien entitled "With Undefined Behavior, Anything is
+Possible". That obviously resonated with Cook, as he pointed out his
+T-shirt—with the title and artwork from the post.
+</p>
+
+<h4>Less danger</h4>
+
+<p>
+He then moved on to some things that kernel developers can do (and are
+doing) to get away from some of the dangers of C. He began with
+variable-length arrays (VLAs), which can be used to overflow the stack to
+access
+data outside of its region. Even if the stack has a guard page, VLAs can
+be used to jump past it to write into other memory, which can then be used
+by some other kind of attack. The C language is "perfectly fine with
+this". It is easy to find uses of VLAs with the <tt>-Wvla</tt> flag, however.
+</p>
+
+<p>
+But it turns out that VLAs are <a href="/Articles/749064/">not just bad
+from a security perspective</a>,
+they are also slow. In a micro-benchmark associated with a <a
+href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=02361bc77888">patch
+removing a VLA</a>, a 13% performance boost came from using a fixed-size
+array. He dug in a bit further and found that much more code is being
+generated to handle a VLA, which explains the speed increase. Since Linus
+Torvalds has <a
+href="https://lore.kernel.org/lkml/CA+55aFzCG-zNmZwX4A2FQpadafLfEzK6CC=qPXydAacU1RqZWA@mail.gmail.com/T/#u">declared
+that VLAs should be removed</a> from the kernel because they cause security
+problems and also slow the kernel down; Cook said "don't use VLAs".
+</p>
+
+<p>
+Another problem area is <tt>switch</tt> statements, in particular where
+there is no <tt>break</tt> for a <tt>case</tt>. That could mean that the
+programmer expects and wants to fall through to the next case or it could
+be that the <tt>break</tt> was simply forgotten. There is a way to get a
+warning from the compiler for fall-throughs, but there needs to be a way to
+mark those that are truly meant to be that way. A special fall-through
+"statement" in the form of a comment is what has been agreed on within the
+static-analysis community. He and others have been going through each of
+the places where there is no <tt>break</tt> to add these comments (or a
+<tt>break</tt>); they
+have "found a lot of bugs this way", he said.
+</p>
+
+<p>
+Uninitialized local variables will generate a warning, but not if the
+variable is passed in by reference. There are some GCC plugins that will
+automatically initialize these variables, but there are also patches for
+both GCC and Clang to provide a compiler option to do so. Neither of those
+is upstream yet, but Torvalds has praised the effort so the kernel would
+likely use the option. An interesting side
+effect that came about while investigating this was a warning he got about
+unreachable code when he
+enabled the auto-initialization. There were two variables declared just
+after a <tt>switch</tt> (and outside of any <tt>case</tt>), where they
+would never be reached.
+</p>
+
+<p>
+Arithmetic overflow is another undefined behavior in C that can cause various
+ problems. GCC can check for signed overflow, which performs well
+(the overhead is in the noise, he said), but adding warning messages for it does grow
+the kernel by 6%; making the overflow abort, instead, only adds 0.1%.
+Clang can check for both signed and unsigned overflow; signed overflow is
+undefined, while unsigned overflow is defined, but often unexpected.
+Marking places where unsigned overflow is expected is needed; it
+would be nice to get those annotations put into the kernel, Cook said.
+</p>
+
+<p>
+Explicit bounds checking is expensive. Doing it for
+<tt>copy_{to,from}_user()</tt> is a less than 1% performance hit, but
+adding it to the <tt>strcpy()</tt> and <tt>memcpy()</tt> families are
+around a 2% hit. Pre-Meltdown that would have been a totally impossible
+performance regression for security, he said; post-Meltdown, since it is
+less than 5%, maybe there is a chance to add this checking.
+</p>
+
+<p>
+Better APIs would help as well. He pointed to the evolution of
+<tt>strcpy()</tt>, through <tt>str<b>n</b>cpy()</tt> and
+<tt>str<b>l</b>cpy()</tt> (each with their own bounds flaws) to
+<tt>str<b>s</b>cpy()</tt>, which seems to be "OK so far". He also mentioned
+<tt>memcpy()</tt> again as a poor API with respect to bounds checking.
+</p>
+
+<p>
+Hardware support for bounds checking is available in the application
+data integrity (ADI) feature for SPARC and is coming for Arm; it may also be
+available for Intel processors at some point. These all use a form of
+"memory tagging", where allocations get a tag that is stored in the
+high-order byte of the address. An offset from the address can be checked
+by the hardware to see if it still falls within the allocated region based
+on the tag.
+</p>
+
+<p>
+Control-flow integrity (CFI) has become more of an issue lately because
+much of what attackers had used in the past has been marked as "no execute"
+so they are turning to using existing code "gadgets" already present in the
+kernel by hijacking existing indirect function calls. In C, you can just call
+pointers without regard to the type as it just treats them as an
+address to jump to. Clang has a CFI-sanitize feature that enforces the
+function prototype to restrict the calls that can be made. It is done at
+runtime and is not perfect, in part because there are lots of functions in
+the kernel that take one unsigned long parameter and return an unsigned long.
+</p>
+
+<p>
+Attacks on CFI have both a "forward edge", which is what CFI sanitize
+tries to handle, and a "backward edge" that comes from manipulating the stack
+values, the return address in particular. Clang has two methods available
+to prevent the stack manipulation. The first is the "safe stack", which
+puts various important items (e.g. "safe" variables, register spills, and
+the return address) on a separate stack. Alternatively, the "shadow stack"
+feature creates a separate stack just for return addresses.
+</p>
+
+<p>
+One problem with these other stacks is that they are still writable, so if
+an attacker can find them in memory, they can still perform their attacks.
+Hardware-based protections, like Intel's Control-Flow Enforcement
+Technology (CET), <a href="/Articles/758245/">provides a read-only shadow
+call stack</a> for return addresses. Another hardware protection is <a
+href="/Articles/718888/">pointer authentication</a> for Arm, which adds a
+kind of encrypted tag to the return address that can be verified before it
+is used.
+</p>
+
+<h4>Status and challenges</h4>
+
+<p>
+Cook then went through the current status of handling these different
+problems in the kernel. VLAs are almost completely gone, he said, just a
+few remain in the crypto subsystem; he hopes those VLAs will be gone by 4.20 (or
+whatever the number of the next kernel release turns out to be). Once that
+happens, he plans to turn on <tt>-Wvla</tt> for the kernel build so that
+none creep back in.
+</p>
+
+<p>
+There has been steady progress made on marking fall-through cases in
+<tt>switch</tt> statements. Only 745 remain to be handled of the 2311 that
+existed when this work started; each one requires scrutiny to determine
+what the author's intent is. Auto-initialized local variables can be done
+using compiler plugins, but that is "not quite what we want", he said.
+More compiler support would be helpful there. For arithmetic overflow, it
+would be nice to see GCC get support for the unsigned case, but memory
+allocations are now doing explicit overflow checking at this point.
+</p>
+
+<p>
+Bounds checking has seen some "crying about performance hits", so we are
+waiting impatiently for hardware support, he said. CFI forward-edge
+protection needs <a href="/Articles/744507/">link-time optimization</a>
+(LTO) support for Clang in the kernel, but it is currently working on
+Android. For backward-edge mitigation, the Clang shadow call stack is
+working on Android, but we are impatiently waiting for hardware support for
+that too.
+</p>
+
+<p>
+There are a number of challenges in doing security development for the
+kernel, Cook said. There are cultural boundaries due to conservatism
+within the kernel community; that requires patiently working and reworking
+features in order to get them upstream. There are, of course, technical
+challenges because of the complexity of security changes; those kinds of
+problems can be solved. There are also resource limitations in terms of
+developers, testers, reviewers, and so on. KSPP and the other kernel
+security developers are still making that "slow but steady" progress.
+</p>
+
+<p>
+Cook's <a href="https://outflux.net/slides/2018/lss/danger.pdf">slides
+[PDF]</a> are available for interested readers; before long, there should
+be a video available of the talk as well.
+
+<p>
+[I would like to thank LWN's travel sponsor, the Linux Foundation, for
+travel assistance to attend the Linux Security Summit in Vancouver.]
+<p><a href="/Articles/763641/#Comments">Comments (70 posted)</a>
+<p>
+<a name="763106"></a><h2 class="SummaryHL"><a href="/Articles/763106/">The second half of the 4.19 merge window</a></h2>
+
+<div class="FeatureByline">
+ By <b>Jonathan Corbet</b><br>August 26, 2018
+ </div>
+By the time Linus Torvalds <a href="/Articles/763497/">released
+4.19-rc1</a> and closed
+the merge window for this development cycle, 12,317 non-merge
+changesets had found their way into the mainline; about 4,800 of those
+landed after <a href="/Articles/762566/">last week's summary</a> was
+written. As tends to be the case
+late in the merge window, many of those changes were fixes for the bigger
+patches that went in early, but there were also a number of new features
+added. Some of the more significant changes include:
+<br clear="all">
+<p>
+
+<h4>Core kernel</h4>
+<p>
+<ul class="spacylist">
+
+<li> The full set of patches adding <a
+ href="/Articles/761118/">control-group awareness to the out-of-memory
+ killer</a> has <i>not</i> been merged due to ongoing disagreements,
+ but one piece of it has: there is a new <tt>memory.oom.group</tt>
+ control knob that will cause all processes within a control group to
+ be killed in an out-of-memory situation.
+<li> A new set of protections has been added to prevent an attacker from
+ fooling a program into writing to an existing file or FIFO. An open
+ with the <tt>O_CREAT</tt> flag to a file or FIFO in a world-writable,
+ sticky
+ directory (e.g. <tt>/tmp</tt>) will fail if the owner of the opening
+ process is not the owner of either the target file or the containing
+ directory. This behavior, disabled by default, is controlled by the
+ new <tt>protected_regular</tt> and <tt>protected_fifos</tt> sysctl
+ knobs.
+
+</ul>
+
+<h4>Filesystems and block layer</h4>
+<p>
+<ul class="spacylist">
+
+<li> The dm-integrity device-mapper target can now use a separate device
+ for metadata storage.
+<li> EROFS, the "enhanced read-only filesystem", has been added to the
+ staging tree. It is "<span>a lightweight read-only file system with
+ modern designs (eg. page-sized blocks, inline xattrs/data, etc.) for
+ scenarios which need high-performance read-only requirements,
+ eg. firmwares in mobile phone or LIVECDs</span>"
+<li> The new "metadata copy-up" feature in overlayfs will avoid copying a
+ file's contents to the upper layer on a metadata-only change. See <a
+ href="https://git.kernel.org/linus/d5791044d2e5749ef4de84161cec5532e2111540">this
+ commit</a> for details.
+
+</ul>
+<p>
+
+<h4>Hardware support</h4>
+<p>
+<ul class="spacylist">
+
+<li> <b>Graphics</b>:
+ Qualcomm Adreno A6xx GPUs.
+
+<li> <b>Industrial I/O</b>:
+ Spreadtrum SC27xx series PMIC analog-to-digital converters,
+ Analog Devices AD5758 digital-to-analog converters,
+ Intersil ISL29501 time-of-flight sensors,
+ Silicon Labs SI1133 UV index/ambient light sensor chips, and
+ Bosch Sensortec BME680 sensors.
+
+
+<li> <b>Miscellaneous</b>:
+ Generic ADC-based resistive touchscreens,
+ Generic ASIC devices via the Google <a
+ href="/ml/linux-kernel/20180630000253.70103-1-sque@chromium.org/">Gasket
+ framework</a>,
+ Analog Devices ADGS1408/ADGS1409 multiplexers,
+ Actions Semi Owl SoCs DMA controllers,
+ MEN 16Z069 watchdog timers,
+ Rohm BU21029 touchscreen controllers,
+ Cirrus Logic CS47L35, CS47L85, CS47L90, and CS47L91 codecs,
+ Cougar 500k gaming keyboards,
+ Qualcomm GENI-based I2C controllers,
+ Actions Semiconductor Owl I2C controllers,
+ ChromeOS EC-based USBPD chargers, and
+ Analog Devices ADP5061 battery chargers.
+
+<li> <b>USB</b>:
+ Nuvoton NPCM7XX on-chip EHCI USB controllers,
+ Broadcom Stingray PCIe PHYs, and
+ Renesas R-Car generation 3 PCIe PHYs.
+
+<li> There is also a new subsystem for the abstraction of GNSS (global
+ navigation satellite systems — GPS, for example) receivers in the
+ kernel. To date, such devices have been handled with an abundance of
+ user-space drivers; the hope is to bring some order in this area.
+ Support for u-blox and SiRFstar receivers has been added as well.
+
+</ul>
+
+
+<p>
+<h4>Kernel internal</h4>
+<p>
+<ul class="spacylist">
+
+<li> The <tt>__deprecated</tt> marker, used to mark interfaces that should
+ no longer be used, has been deprecated and removed from the kernel
+ entirely. <a
+ href="https://git.kernel.org/linus/771c035372a036f83353eef46dbb829780330234">Torvalds
+ said</a>: "<span>They are not useful. They annoy
+ everybody, and nobody ever does anything about them, because it's
+ always 'somebody elses problem'. And when people start thinking that
+ warnings are normal, they stop looking at them, and the real warnings
+ that mean something go unnoticed.</span>"
+<li> The minimum version of GCC required by the kernel has been moved up to
+ 4.6.
+
+</ul>
+<p>
+
+There are a couple of significant changes that failed to get in this time
+around, including the <a
+href="/Articles/745073/">XArray</a> data structure. The patches are
+thought to be ready, but they had the bad luck to be based on a tree that
+failed to be merged for other reasons, so Torvalds <a
+href="/ml/linux-kernel/CA+55aFxFjAmrFpwQmEHCthHOzgidCKnod+cNDEE+3Spu9o1s3w@mail.gmail.com/">didn't
+even look at them</a>. That, in turn, blocks another set of patches intended to
+enable migration of slab-allocated objects.
+<p>
+The other big deferral is the <a href="/Articles/759499/">new system-call
+API for filesystem mounting</a>. Despite ongoing <a
+href="/Articles/762355/">concerns</a> about what happens when the same
+low-level device is mounted multiple times with conflicting options, Al
+Viro sent <a
+href="/ml/linux-fsdevel/20180823223145.GK6515@ZenIV.linux.org.uk/">a pull
+request</a> to send this work upstream. The ensuing discussion made it
+clear that there is still not a consensus in this area, though, so it seems
+that this work has to wait for another cycle.
+<p>
+Assuming all goes well, the kernel will stabilize over the coming weeks and
+the final 4.19 release will happen in mid-October.
+<p><a href="/Articles/763106/#Comments">Comments (1 posted)</a>
+<p>
+<a name="763603"></a><h2 class="SummaryHL"><a href="/Articles/763603/">Measuring (and fixing) I/O-controller throughput loss</a></h2>
+
+<div class="GAByline">
+ <p>August 29, 2018</p>
+ <p>This article was contributed by Paolo Valente</p>
+ </div>
+<p>Many services, from web hosting and video streaming to cloud storage,
+need to move data to and from storage. They also often require that each per-client
+I/O flow be guaranteed a non-zero amount of bandwidth and a bounded latency. An
+expensive way to provide these guarantees is to over-provision
+storage resources, keeping each resource underutilized, and thus
+have plenty of bandwidth available for the few I/O flows dispatched to
+each medium. Alternatively one can use an I/O controller. Linux provides
+two mechanisms designed to throttle some I/O streams to allow others to
+meet their bandwidth and latency requirements. These mechanisms work, but
+they come at a cost: a loss of as much as 80% of total available I/O
+bandwidth. I have run some tests to demonstrate this problem; some
+upcoming improvements to the <a href="/Articles/601799/">bfq I/O
+scheduler</a> promise to improve the situation considerably.
+<p>
+
+<p>Throttling does guarantee control, even on drives that happen to be
+highly utilized but, as will be seen, it has a hard time
+actually ensuring that drives are highly utilized. Even with greedy I/O
+flows, throttling
+easily ends up utilizing as little as 20% of the available speed of a
+flash-based drive.
+
+Such a speed loss may be particularly problematic with lower-end
+storage. On the opposite end, it is also disappointing with
+high-end hardware, as the Linux block I/O stack itself has been
+<a href="/Articles/552904">redesigned from the ground up</a> to fully utilize the
+high speed of modern, fast storage. In
+addition, throttling fails to guarantee the expected bandwidths if I/O
+contains both reads and writes, or is sporadic in nature.
+
+<p>On the bright side, there now seems to be an effective alternative for
+controlling I/O: the proportional-share policy provided by the bfq I/O
+scheduler. It enables nearly 100% storage bandwidth utilization,
+at least with some of the workloads that are problematic for
+throttling. An upcoming version of bfq may be able to
+achieve this result with almost all workloads. Finally, bfq
+guarantees bandwidths with all workloads. The current limitation of
+bfq is that its execution overhead becomes significant at speeds above
+400,000 I/O operations per second on commodity CPUs.
+
+<p>Using the bfq I/O scheduler, Linux can now guarantee
+low latency to lightweight flows containing sporadic, short I/O. No
+throughput issues arise, and no configuration is required. This
+capability benefits important, time-sensitive tasks, such as
+video or audio streaming, as well as executing commands or starting
+applications.
+
+Although benchmarks are not available yet, these guarantees might also be
+provided by the newly proposed <a href="/Articles/758963/">I/O latency
+controller</a>. It allows administrators to set target latencies for I/O
+requests originating from each group of processes, and favors the
+groups with the lowest target latency.
+
+<h4>The testbed</h4>
+
+<p>I ran the tests with an ext4 filesystem mounted on a PLEXTOR
+PX-256M5S SSD, which features a peak rate of ~160MB/s with random I/O,
+and of ~500MB/s with sequential I/O. I used blk-mq, in Linux
+4.18. The system was equipped with a 2.4GHz Intel Core i7-2760QM
+CPU and 1.3GHz DDR3 DRAM. In such a system, a single thread doing
+synchronous reads reaches a throughput of 23MB/s.
+
+<p>
+For the purposes of these tests, each process is considered to be in one of
+two groups, termed "target" and "interferers".
+A target is a single-process, I/O-bound group whose I/O is focused on. In
+particular, I measure the I/O throughput enjoyed by this group to get
+the minimum bandwidth delivered to the group.
+An interferer is single-process group whose role is to generate
+additional I/O that interferes with the I/O of the target.
+The tested workloads contain one target and multiple interferers.
+
+<p>The single process in each group either reads or writes, through
+asynchronous (buffered) operations, to one file — different from the file read
+or written by any other process — after invalidating the buffer cache
+for the file. I define a reader or writer process as either "random" or
+"sequential", depending on whether it reads or writes its file at random
+positions or sequentially.
+Finally, an interferer is defined as being either "active" or "inactive"
+depending on whether it performs I/O during the test. When an
+interferer is mentioned, it is assumed that the interferer is active.
+
+<p>Workloads are defined so as to try to cover the combinations that, I
+believe, most influence the performance of the storage device and of
+the I/O policies. For brevity, in this article I show results for only
+two groups of workloads:
+<p>
+<ul class="spacylist">
+
+<li> <b>Static sequential</b>: four synchronous sequential readers or four
+ asynchronous sequential writers, plus five inactive interferers.
+
+<li> <b>Static random</b>: four synchronous random readers, all with a block
+ size equal to 4k, plus five inactive interferers.
+</ul>
+
+<p>To create each workload, I considered, for each mix of
+interferers in the group, two possibilities for the target: it could be
+either a random or a sequential synchronous reader.
+
+In <a
+href="http://algogroup.unimore.it/people/paolo/pub-docs/extended-lat-bw-throughput.pdf">a
+longer version of this article [PDF]</a>, you will also find results
+for workloads with varying degrees of I/O randomness, and for
+dynamic workloads (containing sporadic I/O sources). These extra results
+confirm the losses of throughput and I/O control for throttling that
+are shown here.
+
+<h4>I/O policies</h4>
+
+<p>Linux provides two I/O-control mechanisms for guaranteeing (a minimum)
+bandwidth, or at least fairness, to long-lived flows: the throttling
+and proportional-share I/O policies.
+With throttling, one can set a maximum bandwidth limit — "max limit" for
+brevity — for the I/O of each group. Max limits can be used,
+in an indirect way, to provide the service guarantee at the focus of this
+article. For example, to guarantee minimum bandwidths to I/O flows, a group can
+be guaranteed a minimum bandwidth by limiting the maximum bandwidth of
+all the other groups.
+
+<p>Unfortunately, max limits have two drawbacks in terms of
+throughput. First, if some groups do not use their allocated bandwidth,
+that bandwidth cannot be reclaimed by other active groups. Second,
+limits must comply with the worst-case speed of the device, namely,
+its random-I/O peak rate. Such limits will clearly leave a lot of
+throughput unused with workloads that otherwise would drive the
+device to higher throughput levels.
+
+Maximizing throughput is simply not a goal of max limits. So, for
+brevity, test results with max limits are not shown here. You can
+find these results, plus a more detailed description of the above
+drawbacks, in the long version of this article.
+
+<p>Because of these drawbacks, a new, still experimental, low limit
+has been added to the throttling policy. If a group is
+assigned a low limit, then the throttling policy automatically
+limits the I/O of the other groups in such a way to
+guarantee to the group a minimum bandwidth equal to its assigned low
+limit. This new throttling mechanism throttles no group as long as
+every group is getting at least its assigned minimum bandwidth. I tested
+this mechanism, but did not consider the interesting problem
+of guaranteeing minimum bandwidths while, at the same time, enforcing
+maximum bandwidths.
+
+<p>The other I/O policy available in Linux, proportional share,
+provides weighted fairness. Each group is assigned a weight, and should
+receive a portion of the total throughput proportional to its weight.
+This scheme guarantees minimum bandwidths in the same way that low limits do
+in throttling. In particular, it guarantees to each group a minimum
+bandwidth equal to the ratio between the weight of the group, and the
+sum of the weights of all the groups that may be active at the same
+time.
+
+<p>The actual implementation of the proportional-share policy, on a given
+drive, depends on what flavor of the block layer is in use for that
+drive. If the drive is using the legacy block interface, the policy is
+implemented by
+the cfq I/O scheduler. Unfortunately, cfq fails to control
+bandwidths with flash-based storage, especially on drives featuring
+command queueing. This case is not considered in these tests. With
+drives using the multiqueue interface,
+proportional share is implemented by bfq. This is the
+combination considered in the tests.
+
+<p>To benchmark both throttling (low limits) and proportional share, I
+tested, for each workload, the combinations of I/O policies and I/O
+schedulers reported in the table below. In the end, there are three test
+cases for each workload. In addition, for some workloads, I considered two
+versions of bfq for the proportional-share policy.
+
+<blockquote>
+<table class="OddEven">
+<tr>
+<th align="left" width="14%" valign="top">Name </th>
+<th align="left" width="14%" valign="top">I/O policy </th>
+<th align="left" width="14%" valign="top">Scheduler </th>
+<th align="left" width="14%" valign="top">Parameter for target </th>
+<th align="left" width="14%" valign="top">Parameter for each
+of the four active interferers </th>
+<th align="left" width="14%" valign="top">Parameter for each of the five inactive
+interferers </th>
+<th align="left" width="14%" valign="top">Sum of parameters</th>
+</tr>
+<tr>
+<td align="left" width="14%" valign="top">low-none</td>
+<td align="left" width="14%" valign="top">Throttling with low limits</td>
+<td align="left" width="14%" valign="top">none</td>
+<td align="left" width="14%" valign="top">10MB/s</td>
+<td align="left" width="14%" valign="top">10MB/s
+(tot: 40)</td>
+<td align="left" width="14%" valign="top">20MB/s (tot: 100)</td>
+<td align="left" width="14%" valign="top">150MB/s</td>
+</tr>
+<tr>
+<td align="left" width="14%" valign="top">prop-bfq</td>
+<td align="left" width="14%" valign="top">Proportional share</td>
+<td align="left" width="14%" valign="top">bfq</td>
+<td align="left" width="14%" valign="top">300</td>
+<td align="left" width="14%" valign="top">100 (tot: 400)</td>
+<td align="left" width="14%" valign="top">200
+(tot: 1000)</td>
+<td align="left" width="14%" valign="top">1700</td>
+</tr>
+</table>
+</blockquote>
+
+
+
+
+<p>For low limits, I report results with only none as the I/O scheduler,
+because the results are the same with kyber and mq-deadline.
+
+<p>The capabilities of the storage medium and of low limits drove the policy
+configurations. In particular:
+
+<ul class="spacylist">
+
+<li> The configuration of the target and of the active interferers for
+low-none is the one for which low-none provides
+its best possible minimum-bandwidth guarantee to the target: 10MB/s,
+guaranteed if all interferers are readers.
+Results remain the same regardless of the values used for target
+latency and idle time; I set them to 100µs and
+1000µs, respectively, for every group.</li>
+
+<li> Low limits for inactive interferers are set to twice the limits for
+active interferers, to pose greater difficulties to the
+policy.</li>
+
+<li> I chose weights for prop-bfq so as to guarantee about the same
+minimum bandwidth as low-none to the target, in the same
+only-reader worst case as for low-none and to preserve, between
+the weights of active and inactive interferers, the same ratio as
+between the low limits of active and inactive interferers.</li>
+</ul>
+<p>Full details on configurations can be found in the long version of this
+article.
+
+<p>Each workload was run ten times for each policy, plus ten times without
+any I/O control, i.e., with none as I/O scheduler and no I/O policy in
+use. For each run, I measured the I/O throughput of the target (which
+reveals the bandwidth provided to the target), the cumulative I/O
+throughput of the interferers, and the total I/O throughput. These
+quantities fluctuated very little during each run, as well as across
+different runs. Thus in the graphs I report only averages over per-run
+average throughputs. In particular, for the case of no I/O control, I
+report only the total I/O throughput, to give an idea of the throughput
+that can be reached without imposing any control.
+
+<h4>Results</h4>
+
+<p>
+This plot shows throughput results for the simplest group of
+workloads: the static-sequential set.
+
+<blockquote>
+<img src="https://static.lwn.net/images/2018/iocontrol/fig1.png" alt="[Figure 1]" class="photo">
+</blockquote>
+<p>
+
+With a random reader as
+the target against sequential readers as interferers, low-none does
+guarantee the configured low limit to the target. Yet it reaches only a
+low total throughput. The throughput of
+the random reader evidently oscillates around 10MB/s during the test.
+This implies that it is at least slightly below 10MB/s for a significant
+percentage of the time. But when this happens, the low-limit mechanism
+limits the maximum bandwidth of every active group to the low limit set
+for the group, i.e., to just 10MB/s.
+The end result is a total throughput lower than 10% of the throughput
+reached without I/O control.
+<p>
+That said, the high throughput achieved without I/O control is
+obtained by choking the random I/O of the target in favor of
+the sequential I/O of the interferers. Thus, it
+is probably more interesting to compare low-none throughput with the
+throughput reachable while actually guaranteeing 10MB/s to the target.
+The target is a single, synchronous, random reader, which reaches 23MB/s while
+active. So, to guarantee 10MB/s to the target, it is enough to
+serve it for about half of the time, and the interferers for the other
+half. Since the device reaches ~500MB/s with the sequential I/O of the
+interferers, the resulting throughput with this service scheme would be
+(500+23)/2, or about 260MB/s. low-none thus reaches less than 20%
+of the
+total throughput that could be reached while still preserving the target
+bandwidth.
+
+<p>prop-bfq provides the target with a slightly higher throughput than
+low-none. This makes it harder for prop-bfq to reach a high total
+throughput, because prop-bfq serves more random I/O (from the target)
+than low-none. Nevertheless, prop-bfq gets a much higher total
+throughput than low-none. According to the above estimate, this
+throughput is about 90% of the maximum throughput that could be reached,
+for this workload, without violating service guarantees. The reason for
+this good result is that bfq provides an effective implementation of
+the proportional-share service policy. At any time, each active group is
+granted a fraction of the current total throughput, and the sum of these
+fractions is equal to one; so group bandwidths naturally saturate the
+available total throughput at all times.
+
+<p>Things change with the second workload: a random reader against
+sequential writers. Now low-none reaches a much higher total
+throughput than prop-bfq. low-none serves
+much more sequential (write) I/O than prop-bfq because writes somehow
+break the low-limit mechanisms and prevail over the reads of the target.
+Conceivably, this happens because writes tend to both starve reads in
+the OS (mainly by eating all available I/O tags) and to cheat on their
+completion time in the drive. In contrast, bfq is intentionally
+configured to privilege reads, to counter these issues.
+
+<p>In particular, low-none gets an even higher throughput than no
+I/O control at all because it penalizes the random I/O of the target even more
+than the no-controller configuration.
+
+<p>Finally, with the last two workloads, prop-bfq reaches even
+higher total throughput than with the first two. It happens
+because the target also does sequential I/O, and serving sequential
+I/O is much more beneficial for throughput than serving random I/O. With
+these two workloads, the total throughput is, respectively, close to or
+much higher than that reached without I/O control. For the last
+workload, the total throughput is much higher because, differently from
+none, bfq privileges reads over asynchronous writes, and reads yield
+a higher throughput than writes. In contrast, low-none still gets
+lower or much lower throughput than prop-bfq, because of the same
+issues that hinder low-none throughput with the first two workloads.
+
+<p>As for bandwidth guarantees, with readers as interferers (third
+workload), prop-bfq, as expected, gives the target a fraction of the
+total throughput proportional to its weight. bfq approximates
+perfect proportional-share bandwidth distribution among groups doing I/O
+of the same type (reads or writes) and with the same locality
+(sequential or random). With the last workload, prop-bfq gives much
+more throughput to the reader than to all the interferers, because
+interferers are asynchronous writers, and bfq privileges reads.
+
+<p>The second group of workloads (static random), is the one, among all
+the workloads considered, for which prop-bfq performs worst.
+Results are shown below:
+<p>
+<blockquote>
+<img src="https://static.lwn.net/images/2018/iocontrol/fig2.png" alt="[Figure 2]" class="photo">
+</blockquote>
+<p>
+
+This chart
+reports results not only for mainline bfq, but also for an
+improved version of
+bfq which is currently under public testing.
+As can be seen, with only random readers, prop-bfq reaches a
+much lower total throughput than low-none. This happens because of
+the Achilles heel of the bfq I/O scheduler. If the process in service
+does synchronous I/O and has a higher weight than some other process, then, to
+give strong bandwidth guarantees to that process, bfq plugs I/O
+dispatching every time the process temporarily stops issuing
+I/O requests. In this respect, processes actually have differentiated
+weights and do synchronous I/O in the workloads tested. So bfq
+systematically performs I/O plugging for them. Unfortunately, this
+plugging empties the internal queues of the drive, which kills
+throughput with random I/O. And the I/O of all processes in these
+workloads is also random.
+
+<p>The situation reverses with a sequential reader as target. Yet, the most
+interesting results come from the new version of bfq, containing
+small changes to counter exactly the above weakness. This
+version recovers most of the throughput loss with the workload made of
+only random I/O and more; with the second workload, where the target is
+a sequential reader, it reaches about 3.7 times the total throughput of
+low-none.
+<p>
+
+When the main concern is the latency of flows containing short I/O,
+Linux seems now rather high performing, thanks to the bfq I/O
+scheduler and the I/O latency controller. But if the
+requirement is to provide explicit bandwidth guarantees (or just fairness) to
+I/O flows, then one must be ready to give up much or most of the speed of
+the storage media. bfq helps with some workloads, but loses most of
+the throughput with workloads consisting of mostly random
+I/O. Fortunately, there is apparently hope for much better
+performance since an improvement, still under development, seems to
+enable bfq to reach a high throughput with all workloads tested so
+far.
+
+
+
+<p>
+[ I wish to thank Vivek Goyal for enabling me to make this article
+much more fair and sound.]<div class="MakeALink">
+ <table align="right"><tr><td>
+ <form action="/SubscriberLink/MakeLink" method="post">
+ <input type="hidden" name="articleid" value="763603">
+ <input type="submit" value="Send a free link"></form>
+ </td></tr></table>
+ </div>
+ <br clear="all">
+
+<p><a href="/Articles/763603/#Comments">Comments (4 posted)</a>
+<p>
+<a name="763175"></a><h2 class="SummaryHL"><a href="/Articles/763175/">KDE's onboarding initiative, one year later</a></h2>
+
+<div class="GAByline">
+ <p>August 24, 2018</p>
+ <p>This article was contributed by Marta Rybczyńska</p>
+ <hr>
+<a href="/Archives/ConferenceByYear/#2018-Akademy">Akademy</a>
+</div>
+<p>In 2017, the KDE community decided on <a
+href="https://dot.kde.org/2017/11/30/kdes-goals-2018-and-beyond">three
+goals</a>
+to concentrate on for the next few years. One of them was <a
+href="https://phabricator.kde.org/T7116">streamlining the onboarding of new
+contributors</a> (the others were <a
+href="https://phabricator.kde.org/T6831">improving
+usability</a> and <a href="https://phabricator.kde.org/T7050">privacy</a>).
+During <a href="https://akademy.kde.org/">Akademy</a>, the yearly KDE
+conference
+that was held in Vienna in August, Neofytos Kolokotronis shared the status
+of the
+onboarding goal, the work done during the last year, and further plans.
+While it is a complicated process in a project as big and diverse as KDE,
+numerous improvements have been already made.</p>
+
+<p>Two of the three KDE community goals were proposed by relative
+newcomers. Kolokotronis was one of those, having joined the <a
+href="https://community.kde.org/Promo">KDE Promo team</a>
+not long before proposing
+the focus on onboarding. He had previously been involved with <a
+href="https://www.chakralinux.org/">Chakra
+Linux</a>, a distribution based on KDE software. The fact that new
+members of the community proposed strategic goals was also noted in the <a
+href="https://conf.kde.org/en/Akademy2018/public/events/79">Sunday keynote
+by Claudia Garad</a>.</p>
+
+<p>Proper onboarding adds excitement to the contribution process and
+increases retention, he explained. When we look at <a
+href="https://en.wikipedia.org/wiki/Onboarding">the definition of
+onboarding</a>,
+it is a process in which the new contributors acquire knowledge, skills, and
+behaviors so that they can contribute effectively. Kolokotronis proposed
+to see it also as socialization: integration into the project's relationships,
+culture, structure, and procedures.</p>
+
+<p>The gains from proper onboarding are many. The project can grow by
+attracting new blood with new perspectives and solutions. The community
+maintains its health and stays vibrant. Another important advantage of
+efficient onboarding is that replacing current contributors becomes easier
+when they change interests, jobs, or leave the project for whatever reason.
+Finally, successful onboarding adds new advocates to the project.</p>
+
+<h4>Achievements so far and future plans</h4>
+
+<p>The team started with ideas for a centralized onboarding process for the
+whole of KDE. They found out quickly that this would not work because KDE
+is "very decentralized", so it is hard to provide tools and
+procedures that are going to work for the whole project. According to
+Kolokotronis, other characteristics of KDE that impact onboarding are high
+diversity, remote and online teams, and hundreds of contributors in dozens of
+projects and teams. In addition, new contributors already know in which
+area they want to take part and they prefer specific information that will
+be directly useful for them.</p>
+
+<p>So the team changed its approach; several changes have since been proposed
+and implemented. The <a href="https://community.kde.org/Get_Involved">Get
+Involved</a> page, which is expected to be one of the resources new
+contributors read first, has been rewritten. For the <a
+href="https://community.kde.org/KDE/Junior_Jobs">Junior Jobs page</a>, the
+team is
+
+<a href="/Articles/763189/"><img
+src="https://static.lwn.net/images/conf/2018/akademy/NeofytosKolokotronis-sm.jpg" alt="[Neofytos
+Kolokotronis]" title="Neofytos Kolokotronis" class="rthumb"></a>
+
+
+<a
+href="https://phabricator.kde.org/T8686">discussing</a> what the
+generic content for KDE as a whole should be. The team simplified <a
+href="https://phabricator.kde.org/T7646">Phabricator registration</a>,
+which
+resulted in documenting the process better. Another part of the work
+includes the <a href="https://bugs.kde.org/">KDE Bugzilla</a>; it includes,
+for example initiatives to limit the number of
+states of a ticket or remove obsolete products.</p>
+
+<p>The <a href="https://www.plasma-mobile.org/index.html">Plasma Mobile</a>
+team is heavily involved in the onboarding goal. The Plasma Mobile
+developers have simplified their
+development environment setup and created an <a
+href="https://www.plasma-mobile.org/findyourway">interactive "Get
+Involved"</a> page. In addition, the Plasma team changed the way task
+descriptions are written; they now contain more detail, so that it is
+easier to get
+involved. The basic description should be short and clear, and it should include
+details of the problem and possible solutions. The developers try to
+share the list of skills necessary to fulfill the tasks and include clear
+links to the technical resources needed.</p>
+
+<p>Kolokotronis and team also identified a new potential source of
+contributors for KDE: distributions using
+KDE. They have the advantage of already knowing and using the software.
+
+The next idea the team is working on is to make sure that setting up a
+development environment is easy. The team plans to work on this during a
+dedicated sprint this autumn.</p>
+
+<h4>Searching for new contributors</h4>
+
+<p>Kolokotronis plans to search for new contributors at the periphery of the
+project, among the "skilled enthusiasts": loyal users who actually care
+about the project. They "can make wonders", he said. Those
+individuals may be also less confident or shy, have troubles making the
+first step, and need guidance. The project leaders should take that into
+account.</p>
+
+<p>In addition, newcomers are all different. Kolokotronis
+provided a long list of how contributors differ,
+including skills and knowledge, motives and
+interests, and time and dedication. His advice is to "try to find their
+superpower", the skills they have that are missing in the team. Those
+"superpowers" can then be used for the benefit of the project.</p>
+
+<p>If a project does nothing else, he said, it can start with its documentation.
+However, this does not only mean code documentation. Writing down the
+procedures or information about the internal work of the project, like who
+is working on what, is an important part of a project's documentation and helps
+newcomers. There should be also guidelines on how to start, especially
+setting up the development environment.</p>
+
+<p>The first thing the project leaders should do, according to
+Kolokotronis, is to spend time on introducing newcomers to the project.
+Ideally every new contributor should be assigned mentors — more
+experienced members who can help them when needed. The mentors and project
+leaders should find tasks that are interesting for each person. Answering
+an audience question on suggestions for shy new
+contributors, he recommended even more mentoring. It is also very helpful
+to make sure that newcomers have enough to read, but "avoid RTFM", he highlighted. It
+is also easy for a new contributor "to fly away", he said. The solution is
+to keep requesting things and be proactive.</p>
+
+<h4>What the project can do?</h4>
+
+<p>Kolokotronis suggested a number of actions for a project when it wants to
+improve its onboarding. The first step is preparation: the project
+leaders should know the team's and the project's needs. Long-term
+planning is important, too. It is not enough to wait for contributors to
+come — the project should be proactive, which means reaching out to
+candidates, suggesting appropriate tasks and, finally, making people
+available for the newcomers if they need help.</p>
+
+<p>This leads to next step: to be a mentor. Kolokotronis suggests being a
+"great host", but also trying to phase out the dependency on the mentor
+rapidly. "We have
+been all newcomers", he said. It can be intimidating to join an existing
+group. Onboarding creates a sense of belonging which, in turn, increases
+retention.</p>
+
+<p>The last step proposed was to be strategic. This includes thinking about
+the emotions you want newcomers to feel. Kolokotronis explained the
+strategic part with an example. The overall goal is (surprise!) improve
+onboarding
+of new contributors. An intermediate objective might be to keep the
+newcomers after they have made their first commit. If your strategy is to keep them
+confident and proud, you can use different tactics like praise and
+acknowledgment of the work in public. Another useful tactic may be assigning
+simple tasks, according to the skill of the contributor.</p>
+
+<p>To summarize, the most important thing, according to Kolokotronis, is to
+respond quickly and spend time with new contributors. This time should be
+used to explain procedures, and to introduce the people and culture. It is also
+essential to guide first contributions and praise contributor's skill and
+effort.
+Increase the difficulty of tasks over time to keep contributors motivated and
+challenged. And finally, he said,
+"turn them into mentors".</p>
+
+<p>Kolokotronis acknowledges that onboarding "takes time" and "everyone
+complains" about it. However, he is convinced that it is beneficial in the
+long term
+and that it decreases developer turnover.</p>
+
+<h4>Advice to newcomers</h4>
+
+<p>Kolokotronis concluded with some suggestions for newcomers to a
+project. They should try
+to be persistent and to not get discouraged when something goes wrong.
+Building connections from the very beginning is helpful. He suggests
+asking questions as if you were already a member "and things will be fine".
+However, accept criticism if it happens.</p>
+
+<p>One of the next actions of the onboarding team will be to collect
+feedback from newcomers and experienced contributors to see if they agree
+on the ideas and processes introduced so far.</p>
+<p><a href="/Articles/763175/#Comments">Comments (none posted)</a>
+<p>
+<a name="763492"></a><h2 class="SummaryHL"><a href="/Articles/763492/">Sharing and archiving data sets with Dat</a></h2>
+
+<div class="GAByline">
+ <p>August 27, 2018</p>
+ <p>This article was contributed by Antoine Beaupré</p>
+ </div>
+<p><a href="https://datproject.org">Dat</a> is a new peer-to-peer protocol
+that uses some of the concepts of
+<a href="https://www.bittorrent.com/">BitTorrent</a> and Git. Dat primarily
+targets researchers and
+open-data activists as it is a great tool for sharing, archiving, and
+cataloging large data sets. But it can also be used to implement
+decentralized web applications in a novel way.</p>
+
+<h4>Dat quick primer</h4>
+
+<p>Dat is written in JavaScript, so it can be installed with <code>npm</code>, but
+there are <a href="https://github.com/datproject/dat/releases">standalone
+binary builds</a> and
+a <a href="https://docs.datproject.org/install">desktop application</a> (as an AppImage). An <a href="https://datbase.org/">online viewer</a> can
+be used to inspect data for those who do not want to install
+arbitrary binaries on their computers.</p>
+
+<p>The command-line application allows basic operations like downloading
+existing data sets and sharing your own.
+Dat uses a 32-byte hex string that is an <a
+href="https://ed25519.cr.yp.to/">ed25519 public key</a>, which is
+is used to discover and find content on the net.
+For example, this will
+download some sample data:</p>
+
+<pre>
+ $ dat clone \
+ dat://778f8d955175c92e4ced5e4f5563f69bfec0c86cc6f670352c457943666fe639 \
+ ~/Downloads/dat-demo
+</pre>
+
+<p>Similarly, the <code>share</code> command is used to share content. It indexes
+the files in a given directory and creates a new unique address like
+the one above. The <code>share</code>
+command starts a server that uses multiple discovery mechanisms (currently, the <a href="https://en.wikipedia.org/wiki/Mainline_DHT">Mainline Distributed
+Hash Table</a> (DHT), a <a href="https://github.com/mafintosh/dns-discovery">custom DNS server</a>, and
+multicast DNS) to announce the content to its peers. This is how
+another user, armed with that public key, can download that content
+with <code>dat clone</code> or mirror the files continuously with
+<code>dat sync</code>.</p>
+
+<p>So far, this looks a lot like BitTorrent <a href="https://en.wikipedia.org/wiki/Magnet_URI_scheme">magnet links</a> updated
+with 21st century cryptography. But Dat adds revisions on top of that,
+so modifications are automatically shared through the swarm. That is
+important for public data sets as those
+are often dynamic in nature. Revisions also make it possible to use
+<a href="https://blog.datproject.org/2017/10/13/using-dat-for-automatic-file-backups/">Dat as a backup system</a> by saving the data incrementally using an
+<a href="https://github.com/mafintosh/hypercore-archiver">archiver</a>.</p>
+
+<p>While Dat is designed to work on larger data sets, processing them
+for sharing may take a while. For example, sharing the Linux
+kernel source code required about five minutes as Dat worked on
+indexing all of the files. This is comparable to the performance offered by
+<a href="https://ipfs.io/">IPFS</a> and BitTorrent. Data sets with
+more or larger files may take quite a bit more time.
+
+<p>
+One advantage that Dat has over IPFS is that it
+doesn't duplicate the data. When IPFS imports new data, it duplicates
+the files into <code>~/.ipfs</code>. For collections of small files like the
+kernel, this is not a huge problem, but for larger files like videos or
+music, it's a significant limitation. IPFS eventually implemented a
+solution to this <a href="https://github.com/ipfs/go-ipfs/issues/875">problem</a> in the form of the experimental
+<a href="https://github.com/ipfs/go-ipfs/blob/master/docs/experimental-features.md#ipfs-filestore">filestore feature</a>, but it's not enabled by default. Even with
+that feature enabled, though, changes to data sets are not automatically
+tracked. In comparison, Dat operation on dynamic data feels much
+lighter. The downside is that each set needs its own <code>dat share</code>
+process.</p>
+
+<p>Like any peer-to-peer system, Dat needs at least one peer to stay online to
+offer the content, which is impractical for mobile devices. Hosting
+providers like <a href="https://hashbase.io/">Hashbase</a> (which is a <a href="https://github.com/datprotocol/DEPs/blob/master/proposals/0003-http-pinning-service-api.md">pinning service</a> in Dat
+jargon) can help users keep content online without running their own
+<a href="https://docs.datproject.org/server">server</a>. The closest parallel in the traditional web ecosystem
+would probably be content distribution networks (CDN) although pinning
+services are not necessarily geographically distributed and a CDN does
+not necessarily retain a complete copy of a website.</p>
+
+<a href="/Articles/763544/">
+<img src="https://static.lwn.net/images/2018/dat-photoapp-sm.png" border=0 hspace=5 align="right"
+width=300 height=392 alt="[Photo app]" title="Photo app">
+</a>
+
+<p>A web browser called <a href="https://beakerbrowser.com/">Beaker</a>, based on the <a href="https://electronjs.org/">Electron</a> framework,
+can access Dat content natively without going through a pinning
+service. Furthermore, Beaker is essential to get any of the <a
+href="https://github.com/beakerbrowser/explore">Dat
+applications</a> working, as they fundamentally rely on <code>dat://</code> URLs
+to do their magic. This means that Dat applications won't work for
+most users unless they install that special web browser. There is a
+<a href="https://addons.mozilla.org/en-US/firefox/addon/dat-p2p-protocol/">Firefox extension</a> called "<a href="https://github.com/sammacbeth/dat-fox">dat-fox</a>" for people who don't want
+to install yet another browser, but it requires installing a
+<a href="https://github.com/sammacbeth/dat-fox-helper">helper program</a>. The extension will be able to load <code>dat://</code> URLs
+but many applications will still not work. For example, the <a
+href="https://github.com/beakerbrowser/dat-photos-app">photo gallery
+application</a> completely fails with dat-fox.</p>
+
+<p>Dat-based applications look promising from a privacy point of view.
+Because of its peer-to-peer nature, users regain control over where
+their data is stored: either on their own computer, an online server, or
+by a trusted third party. But considering the protocol is not well
+established in current web browsers, I foresee difficulties in
+adoption of that aspect of the Dat ecosystem. Beyond that, it is rather
+disappointing that Dat applications cannot run natively in a web
+browser given that JavaScript is designed exactly for that.</p>
+
+<h4>Dat privacy</h4>
+
+<p>An advantage Dat has over other peer-to-peer protocols like BitTorrent
+is end-to-end encryption. I was originally concerned by the encryption
+design when reading the <a
+href="https://github.com/datproject/docs/raw/master/papers/dat-paper.pdf">academic
+paper [PDF]</a>:</p>
+
+<div class="BigQuote">
+ <p>It is up to client programs to make design decisions around which
+ discovery networks they trust. For example if a Dat client decides
+ to use the BitTorrent DHT to discover peers, and
+ they are searching
+ for a publicly shared Dat key (e.g. a key cited publicly in a
+ published scientific paper) with known contents, then because of the
+ privacy design of the BitTorrent DHT it becomes public knowledge
+ what key that client is searching for.</p>
+</div>
+
+<p>So in other words, to share a secret file with another user, the
+public key is transmitted over a secure side-channel, only to then
+leak during the discovery process. Fortunately, the public Dat key
+is not directly used during discovery as it is <a
+href="https://github.com/datprotocol/DEPs/blob/653e0cf40233b5d474cddc04235577d9d55b2934/proposals/0000-peer-discovery.md#discovery-keys">hashed
+with BLAKE2B</a>. Still, the security model of Dat assumes the public
+key is private, which is a rather counterintuitive concept that might upset
+cryptographers and confuse users who are frequently encouraged to type
+such strings in address bars and search engines as part of the Dat
+experience. There is a <a
+href="https://docs.datproject.org/security">security & privacy FAQ</a>
+in the Dat
+documentation warning about this problem:</p>
+
+<div class="BigQuote">
+ <p>One of the key elements of Dat privacy is that the public key is
+ never used in any discovery network. The public key is hashed,
+ creating the discovery key. Whenever peers attempt to connect to
+ each other, they use the discovery key.</p>
+
+ <p>Data is encrypted using the public key, so it is important that this
+ key stays secure.</p>
+</div>
+
+<p>There are other privacy issues outlined in the
+document; it states that "<span>Dat faces similar privacy risks as
+BitTorrent</span>":</p>
+
+<div class="BigQuote">
+ <p>When you download a dataset, your IP address is exposed to the users
+ sharing that dataset. This may lead to honeypot servers collecting
+ IP addresses, as we've seen in Bittorrent. However, with dataset
+ sharing we can create a web of trust model where specific
+ institutions are trusted as primary sources for datasets,
+ diminishing the sharing of IP addresses.</p>
+</div>
+
+<p>A Dat blog post refers to this issue as <a href="https://blog.datproject.org/2016/12/12/reader-privacy-on-the-p2p-web/">reader privacy</a> and it
+is, indeed, a sensitive issue in peer-to-peer networks. It is how
+BitTorrent users are discovered and served scary verbiage from lawyers,
+after all. But Dat makes this a little better because, to join a swarm,
+you must know what you are looking for already, which means peers who
+can look at swarm activity only include users who know the secret
+public key. This works well for secret content, but for larger, public
+data sets, it is a real problem; it is why the Dat project has <a
+href="https://blog.datproject.org/2017/12/10/dont-ship/">avoided
+creating a Wikipedia mirror</a> so far.</p>
+
+<p>I found another privacy issue that is not documented in the security FAQ
+during my review of the protocol. As mentioned earlier,
+the <a href="https://github.com/datprotocol/DEPs/pull/7">Dat discovery
+protocol</a> routinely
+ phones home to DNS servers operated by the Dat project.
+This implies that the default discovery servers (and an
+attacker watching over their traffic) know who is publishing or seeking
+content, in essence discovering the "social network" behind Dat. This
+discovery mechanism can be disabled in clients, but a similar privacy
+issue applies to the DHT as well, although that is distributed so it
+doesn't require trust of the Dat project itself.</p>
+
+<p>Considering those aspects of the protocol, privacy-conscious users
+will probably want to use Tor or other anonymization techniques to
+work around those concerns.</p>
+
+<h4>The future of Dat</h4>
+
+<p><a href="https://blog.datproject.org/2017/06/01/dat-sleep-release/">Dat 2.0 was released in June 2017</a> with performance improvements and
+protocol changes. <a href="https://github.com/datprotocol/DEPs">Dat
+Enhancement Proposals</a> (DEPs) guide the project's
+future development; most work is currently geared toward
+implementing the draft "<a href="https://github.com/datprotocol/DEPs/blob/master/proposals/0008-multiwriter.md">multi-writer proposal</a>" in
+<a href="https://github.com/mafintosh/hyperdb">HyperDB</a>. Without
+multi-writer support, only the
+original publisher of a Dat can modify it. According to Joe Hand,
+co-executive-director of <a href="https://codeforscience.org/">Code for Science & Society</a> (CSS) and
+Dat core developer, in an IRC chat, "supporting multiwriter is a big requirement for lots
+of folks". For example, while Dat might allow Alice to share her
+research results with Bob, he cannot modify or contribute back to those
+results. The multi-writer extension allows for Alice to assign trust
+to Bob so he can have write access to the data.
+
+<p>
+Unfortunately, the
+current proposal doesn't solve the "<span>hard problems</span>" of
+"<span>conflict merges
+and secure key distribution</span>". The former will be worked out through
+user interface tweaks, but the latter is a classic problem that security
+projects have typically trouble finding
+solutions for—Dat is no exception. How will Alice securely trust
+Bob? The OpenPGP web of trust? Hexadecimal fingerprints read over the
+phone? Dat doesn't provide a magic solution to this problem.</p>
+
+<p>Another thing limiting adoption is that Dat is not packaged in any
+distribution that I could find (although I <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=890565">requested it in
+Debian</a>) and, considering the speed of change of the JavaScript
+ecosystem, this is unlikely to change any time soon. A <a
+href="https://github.com/datrs">Rust
+implementation</a> of the Dat protocol has started, however, which
+might be easier to package than the multitude of <a href="https://nodejs.org/en/">Node.js</a>
+modules. In terms of mobile device support, there is an experimental
+Android web browser with Dat support called <a href="https://bunsenbrowser.github.io/#!index.md">Bunsen</a>, which somehow
+doesn't run on my phone. Some adventurous users have successfully run Dat
+in <a href="https://termux.com/">Termux</a>. I haven't found an app
+running on iOS at this
+point.</p>
+
+<p>Even beyond platform support, distributed protocols like Dat have a
+tough slope to climb against the virtual monopoly of more centralized
+protocols, so it remains to be seen how popular those tools will
+be. Hand says Dat is supported by multiple non-profit
+organizations. Beyond CSS, <a href="https://bluelinklabs.com/">Blue Link Labs</a> is working on the
+Beaker Browser as a self-funded startup and a grass-roots
+organization, <a href="https://www.digital-democracy.org/">Digital Democracy</a>, has contributed to the
+project. The <a href="https://archive.org">Internet Archive</a> has <a href="https://blog.archive.org/2018/06/05/internet-archive-code-for-science-and-society-and-california-digital-library-to-partner-on-a-data-sharing-and-preservation-pilot-project/">announced a collaboration</a>
+between itself, CSS, and the California Digital Library to launch a pilot
+project to see "<span>how members of a cooperative, decentralized
+network can leverage shared services to ensure data preservation while
+reducing storage costs and increasing replication counts</span>".
+
+<p>
+Hand said
+adoption in academia has been "slow but steady" and that the <a href="https://github.com/codeforscience/Dat-in-the-Lab">Dat in
+the Lab project</a> has helped identify areas that could help
+researchers adopt the project. Unfortunately, as is the case with many
+free-software projects, he said that "our team is definitely a bit
+limited on bandwidth to push for bigger adoption". Hand
+said that the project received a grant from <a
+href="https://www.mozilla.org/en-US/moss/">Mozilla Open Source
+Support</a> to improve its documentation, which will be a big help.</p>
+
+<p>Ultimately, Dat suffers from a problem common to all peer-to-peer
+applications, which is naming. Dat addresses are not exactly
+intuitive: humans do not remember strings of 64 hexadecimal characters
+well. For this, Dat took a <a href="https://github.com/datprotocol/DEPs/blob/master/proposals/0005-dns.md">similar approach</a> to IPFS by using
+DNS <tt>TXT</tt> records and <code>/.well-known</code> URL paths to bridge existing,
+human-readable names with Dat hashes. So this sacrifices a part of the
+decentralized nature of the project in favor of usability.</p>
+
+<p>I have tested a lot of distributed protocols like Dat in the past and
+I am not sure Dat is a clear winner. It certainly has advantages over
+IPFS in terms of usability and resource usage, but the lack of
+packages on most platforms is a big limit to adoption for most
+people. This means it will be difficult to share content with my
+friends and family with Dat anytime soon, which would probably be my
+primary use case for the project. Until the protocol reaches the wider
+adoption that BitTorrent has seen in terms of platform support, I will
+probably wait before switching everything over to this
+promising project.</p>
+<p><a href="/Articles/763492/#Comments">Comments (11 posted)</a>
+<p>
+<p>
+<b>Page editor</b>: Jonathan Corbet<br>
+<h2>Inside this week's LWN.net Weekly Edition</h2>
+<ul>
+<li> <a href="/Articles/763254/">Briefs</a>: OpenSSH 7.8; 4.19-rc1; Which stable?; Netdev 0x12; Bison 3.1; Quotes; ...
+ <li> <a href="/Articles/763255/">Announcements</a>: Newsletters; events; security updates; kernel patches; ...
+ </ul>
+<b>Next page</b>:
+ <a href="/Articles/763254/">Brief items>></a><br>
+
+</div> <!-- ArticleText -->
+</div>
+<div class="lwn-u-1 pure-u-md-1-6 not-print">
+<div id="azk93271_right_zone"></div>
+</div>
+</div> <!-- pure-grid -->
+
+ <br clear="all">
+ <center>
+ <P>
+ <font size="-2">
+ Copyright © 2018, Eklektix, Inc.<BR>
+
+ Comments and public postings are copyrighted by their creators.<br>
+ Linux is a registered trademark of Linus Torvalds<br>
+ </font>
+ </center>
+
+ <script type="text/javascript">
+ var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
+ document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
+ </script>
+ <script type="text/javascript">
+ try {
+ var pageTracker = _gat._getTracker("UA-2039382-1");
+ pageTracker._trackPageview();
+ } catch(err) {}</script>
+
+ </body></html>
+
\ No newline at end of file
git://git.nikiroo.be / gofetch.git / blobdiff