wdtz / Will Dietzhttps://wdtz.org/2019-02-04T21:06:00-06:00“Shipping Software as LLVM IR” @ LLVM Developer’s Meeting 20162019-02-04T21:06:00-06:002019-02-04T21:06:00-06:00Will Dietztag:wdtz.org,2019-02-04:/shipping-software-as-llvm-ir-llvm-developers-meeting-2016.html<p>Around two years ago I attended the 2016 <span class="caps">LLVM</span> Developer’s Meeting
and led a “Birds of a Feather” (BoF) discussion on the topic
of shipping software as <span class="caps">LLVM</span> <span class="caps">IR</span>. The objective was to bring out
all the parties who are clearly doing so already or interested in doing so
and to try to find ways to work together.</p>
<p>I’ll hopefully write more about this in the future,
but for now here’s a short post about this event as well as
a link to the slides used for the discussion.</p>
<p></p><p>Around two years ago I attended the 2016 <span class="caps">LLVM</span> Developer’s Meeting
and led a “Birds of a Feather” (BoF) discussion on the topic
of shipping software as <span class="caps">LLVM</span> <span class="caps">IR</span>. The objective was to bring out
all the parties who are clearly doing so already or interested in doing so
and to try to find ways to work together.</p>
<p>I’ll hopefully write more about this in the future,
but for now here’s a short post about this event as well as
a link to the slides used for the discussion.</p>
<p></p>
<div class="section" id="slides">
<h2>Slides</h2>
<p>The <a class="reference external" href="https://wdtz.org/files/bof-2016.pdf">slides</a> <a class="footnote-reference" href="#id2" id="id3">[1]</a> worked well, and I should have posted them long ago.</p>
</div>
<div class="section" id="thanks">
<h2>Thanks</h2>
<p>The BoF was very well attended and I would likely have been overwhelmed
by all the ideas and suggestions were it not for a number of
fellow <span class="caps">UIUC</span> folks (all part of the <span class="caps">ALLVM</span> team) who stepped up to
take notes, handle the whiteboard, and help guide and answer questions.</p>
<p>This was also an opportunity to tell the community about <span class="caps">ALLVM</span>,
which was well-enough received for a project still underway :).</p>
</div>
<div class="section" id="references">
<h2>References</h2>
<div class="footnote-table">
<table class="footnote" id="id2"><tr>
<td class="label">[1]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id3">↩</a></td>
<td><a class="reference external" href="https://wdtz.org/files/bof-2016.pdf">https://wdtz.org/files/bof-2016.pdf</a></td>
</tr></table>
</div></div>
A few more ALLVM tidbits are now available2019-02-04T19:00:00-06:002019-02-04T19:00:00-06:00Will Dietztag:wdtz.org,2019-02-04:/a-few-more-allvm-tidbits-are-now-available.html<p>Posted on the <span class="caps">ALLVM</span> github organization,
the <a class="reference external" href="https://github.com/allvm/allvm-meta">allvm-meta</a> repository is now available—
and because naming is hard it ended up being
the repository with perhaps the least “meta” content.</p>
<p>Even so I’m excited to make this public
and will give a brief overview below.</p>
<p></p><p>Posted on the <span class="caps">ALLVM</span> github organization,
the <a class="reference external" href="https://github.com/allvm/allvm-meta">allvm-meta</a> <a class="footnote-reference" href="#id1" id="id2">[1]</a> repository is now available—
and because naming is hard it ended up being
the repository with perhaps the least “meta” content.</p>
<p>Even so I’m excited to make this public
and will give a brief overview below.</p>
<p></p>
<div class="section" id="allvm-logo">
<h2><span class="caps">ALLVM</span> Logo</h2>
<p>Based on the <span class="caps">LLVM</span> logo, this contribution was made by the
especially talented <a class="reference external" href="https://twitter.com/rxwei">Richard Wei</a> <a class="footnote-reference" href="#id3" id="id4">[2]</a> and the resulting graphics
have been used in many places and I’m really thankful for the work.</p>
<p>I’m fond of it and you can find it on the <a class="reference external" href="http://allvm.org"><span class="caps">ALLVM</span> website</a> <a class="footnote-reference" href="#id5" id="id6">[3]</a>,
on my slide templates, the <a class="reference external" href="https://github.com/allvm"><span class="caps">ALLVM</span> GitHub Organization</a> <a class="footnote-reference" href="#id7" id="id8">[4]</a>,
and more places.</p>
<p>There’s even a favicon for use on our <a class="reference external" href="https://nixos.org/hydra">Hydra</a> <a class="footnote-reference" href="#id9" id="id10">[5]</a> <span class="caps">ALLVM</span> <span class="caps">CI</span> server—
not publicly available yet because disk failures complicated the migration
but is likely to be revived on new public hosting soon,
as more of the <span class="caps">ALLVM</span> bits go public.</p>
</div>
<div class="section" id="llvm-jit-in-the-bsd-kernel">
<h2><span class="caps">LLVM</span> <span class="caps">JIT</span> in the <span class="caps">BSD</span> Kernel</h2>
<p><span class="caps">ALLVM</span> researches systems where <span class="caps">ALL</span> code is available as <span class="caps">IR</span>,
and in the “big picture” plan this included the kernel as well as userspace
with the compiler as a first-class component of the <span class="caps">OS</span>.</p>
<p>In this little code dump are most of the bits I used for my exploratory
blitz towards getting an <span class="caps">LLVM</span> <span class="caps">JIT</span> as close to the metal as I could
manage in a few weeks.</p>
<p>Unfortunately some of the patches for uclibc, uclibc++, and <span class="caps">LLVM</span>
have been lost— but when I get a free weekend I think they’re
in an old backup but it’ll take some work to dig through that.</p>
<p>Even so this is the work that caused the <a class="reference external" href="https://wdtz.org/kernel-hacking-my-very-own-bsod.html">fun <span class="caps">BSOD</span></a> <a class="footnote-reference" href="#id11" id="id12">[6]</a> I posted
about at the time: as I recall I did something like scribbled
<span class="caps">LLVM</span> <span class="caps">IR</span> all over system memory due to a silly mistake.</p>
<p>My solution was to shove enough of an <span class="caps">LLVM</span> <span class="caps">JIT</span> and hacked “runtime” support
into a bsd kernel module that hopefully the bootloader or early
kernel init could host.</p>
<p>This worked well enough that I was able to point it at bitcode
for the FreeBSD kernel and successfully codegen the result.</p>
<p>It couldn’t quite execute the result, although I <span class="caps">THINK</span> that
was largely a matter of rejiggering a number of things to
pivot into it gracefully— not because the code was wrong.</p>
<p>I’ll hopefully find those pieces soon,
and might not be ready for exploring but it’s
at least in a better place than it was :).</p>
<p>I wrote an email to the <span class="caps">ALLVM</span> co-founders
my advisor Vikram and <a class="reference external" href="http://quetzalcoatal.blogspot.com/">Joshua Cranmer</a> <a class="footnote-reference" href="#id13" id="id14">[7]</a>,
which is a nice piece of “history”
and the early excitement of the project!</p>
<p>You’ll find it in the allvm-meta repo,
it describes my goals and status better
than the summary above.</p>
<p>Here: <a class="reference external" href="https://raw.githubusercontent.com/allvm/allvm-meta/master/bsd_kernel/ALLVM-Status-update"><span class="caps">ALLVM</span> <span class="caps">JIT</span> Status Email</a> <a class="footnote-reference" href="#id15" id="id16">[8]</a></p>
</div>
<div class="section" id="references">
<h2>References</h2>
<div class="footnote-table">
<table class="footnote" id="id1"><tr>
<td class="label">[1]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id2">↩</a></td>
<td><a class="reference external" href="https://github.com/allvm/allvm-meta">https://github.com/allvm/allvm-meta</a></td>
</tr></table>
<table class="footnote" id="id3"><tr>
<td class="label">[2]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id4">↩</a></td>
<td><a class="reference external" href="https://twitter.com/rxwei">https://twitter.com/rxwei</a></td>
</tr></table>
<table class="footnote" id="id5"><tr>
<td class="label">[3]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id6">↩</a></td>
<td><a class="reference external" href="http://allvm.org">http://allvm.org</a></td>
</tr></table>
<table class="footnote" id="id7"><tr>
<td class="label">[4]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id8">↩</a></td>
<td><a class="reference external" href="https://github.com/allvm">https://github.com/allvm</a></td>
</tr></table>
<table class="footnote" id="id9"><tr>
<td class="label">[5]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id10">↩</a></td>
<td><a class="reference external" href="https://nixos.org/hydra">https://nixos.org/hydra</a></td>
</tr></table>
<table class="footnote" id="id11"><tr>
<td class="label">[6]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id12">↩</a></td>
<td><a class="reference external" href="https://wdtz.org/kernel-hacking-my-very-own-bsod.html">{filename}../technical/kernel_wut.rst</a></td>
</tr></table>
<table class="footnote" id="id13"><tr>
<td class="label">[7]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id14">↩</a></td>
<td><a class="reference external" href="http://quetzalcoatal.blogspot.com/">http://quetzalcoatal.blogspot.com/</a></td>
</tr></table>
<table class="footnote" id="id15"><tr>
<td class="label">[8]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id16">↩</a></td>
<td><a class="reference external" href="https://raw.githubusercontent.com/allvm/allvm-meta/master/bsd_kernel/ALLVM-Status-update">https://raw.githubusercontent.com/allvm/allvm-meta/master/bsd_kernel/<span class="caps">ALLVM</span>-Status-update</a></td>
</tr></table>
</div></div>
efisame: Next Boot, Same as This Boot2015-12-10T16:18:00-06:002015-12-10T16:18:00-06:00Will Dietztag:wdtz.org,2015-12-10:/efisame-next-boot-same-as-this-boot.html<p><a class="reference external" href="https://en.wikipedia.org/wiki/Multi-booting">Multi-booting</a> is a widely-used practice that enables a user
to select the <span class="caps">OS</span> best suited for their current task.
For example, I dual-boot Windows and Linux on my laptop
which allows me to get the best of both worlds on the same machine.</p>
<p>I’ve encountered a minor pain point in this setup however,
largely due to the way I tend to use my machine: once I boot
a particular <span class="caps">OS</span> I continue to use that same <span class="caps">OS</span> repeatedly and
expect reboots/shutdown/startup to continue to use the last
<span class="caps">OS</span> I booted. This is especially true of reboots, for
example after installing system updates.</p>
<p>Traditionally I addressed this by setting the default boot
to my most-frequently-used <span class="caps">OS</span>, but really that is just
optimizing for the common case and not solving the issue.
Additionally, I’ve recently started using Windows more and
am annoyed having to remind my machine what <span class="caps">OS</span> I’m using.</p>
<p>So, today I sat down and put together a simple little
utility to fix this properly once and for all.</p>
<p></p><p><a class="reference external" href="https://en.wikipedia.org/wiki/Multi-booting">Multi-booting</a> <a class="footnote-reference" href="#id1" id="id2">[1]</a> is a widely-used practice that enables a user
to select the <span class="caps">OS</span> best suited for their current task.
For example, I dual-boot Windows and Linux on my laptop
which allows me to get the best of both worlds on the same machine.</p>
<p>I’ve encountered a minor pain point in this setup however,
largely due to the way I tend to use my machine: once I boot
a particular <span class="caps">OS</span> I continue to use that same <span class="caps">OS</span> repeatedly and
expect reboots/shutdown/startup to continue to use the last
<span class="caps">OS</span> I booted. This is especially true of reboots, for
example after installing system updates.</p>
<p>Traditionally I addressed this by setting the default boot
to my most-frequently-used <span class="caps">OS</span>, but really that is just
optimizing for the common case and not solving the issue.
Additionally, I’ve recently started using Windows more and
am annoyed having to remind my machine what <span class="caps">OS</span> I’m using.</p>
<p>So, today I sat down and put together a simple little
utility to fix this properly once and for all.</p>
<p></p>
<div class="section" id="introducing-efisame">
<h2>Introducing: <code class="docutils literal">efisame</code></h2>
<p>This <code class="docutils literal">efisame</code> utility is specifically for systems using <a class="reference external" href="https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface"><span class="caps">UEFI</span></a> <a class="footnote-reference" href="#id3" id="id4">[2]</a> boot
manager, which is commonly used on hardware made in the last
few years.</p>
<p>What <code class="docutils literal">efisame</code> does is fairly simple, perfoming its task
by manipulating the <code class="docutils literal">BootNext</code> and <code class="docutils literal">BootCurrent</code> <span class="caps">EFI</span>
variables.
The <code class="docutils literal">BootNext</code> variable indicates to the <span class="caps">EFI</span> boot manager
what to boot from by default for the next boot only.
<code class="docutils literal">BootCurrent</code> is read-only and indicates the entry used
to boot the current system.
Together, these form our solution: <code class="docutils literal">efisame</code> tool sets
<code class="docutils literal">BootNext</code> to have the value of <code class="docutils literal">BootCurrent</code>.
It wraps up with a sanity check for good measure, but
otherwise that’s all there is to it.</p>
</div>
<div class="section" id="running-efisame-at-boot">
<h2>Running <code class="docutils literal">efisame</code> at Boot</h2>
<p>In order to solve the problem described, <code class="docutils literal">efisame</code>
needs to execute every time the system boots.
Any method works here, but since my laptop uses systemd
I created a systemd service file which simply runs the
program on system startup.</p>
<p>Finally, I changed the default <span class="caps">EFI</span> boot entry to be Windows
and altogether I now have a system that consistently boots
into the last system I manually indicated I wanted to use.</p>
</div>
<div class="section" id="available-now">
<h2>Available Now</h2>
<p>You can get a copy of the utility on the <a class="reference external" href="https://github.com/dtzWill/efi-same">efisame github</a> <a class="footnote-reference" href="#id5" id="id6">[3]</a>,
complete with instructions and the systemd service file.</p>
<p>Let me know if you find it useful, or if you have any
problems or questions.</p>
<p>Enjoy!</p>
</div>
<div class="section" id="references">
<h2>References</h2>
<div class="footnote-table">
<table class="footnote" id="id1"><tr>
<td class="label">[1]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id2">↩</a></td>
<td><a class="reference external" href="https://en.wikipedia.org/wiki/Multi-booting">https://en.wikipedia.org/wiki/Multi-booting</a></td>
</tr></table>
<table class="footnote" id="id3"><tr>
<td class="label">[2]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id4">↩</a></td>
<td><a class="reference external" href="https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface">https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface</a></td>
</tr></table>
<table class="footnote" id="id5"><tr>
<td class="label">[3]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id6">↩</a></td>
<td><a class="reference external" href="https://github.com/dtzWill/efi-same">https://github.com/dtzWill/efi-same</a></td>
</tr></table>
</div></div>
IOC Journal Edition: Highlights2015-12-07T15:33:00-06:002015-12-07T15:33:00-06:00Will Dietztag:wdtz.org,2015-12-07:/ioc-journal-edition-highlights.html<p>I’m excited to announce the publication of the
<a class="reference external" href="http://dx.doi.org/10.1145/2743019">journal version of “Understanding Integer Overflow in C/C++”</a>, appearing
in <a class="reference external" href="http://tosem.acm.org/archive.cfm?id=2852270"><span class="caps">TOSEM</span> Volume 25 Issue 1</a>.
This is an updated and expanded version of our <a class="reference external" href="http://www.cs.utah.edu/~regehr/papers/overflow12.pdf"><span class="caps">ICSE12</span> paper</a> of the same name.
The longer journal format enabled a more thorough treatment of the subject, and
we did our best to take advantage of that opportunity.</p>
<p>Thanks to my co-authors for all their efforts, and especially for seeing this
work through to the end. It’s been a long run and you guys are great.</p>
<p></p><p>I’m excited to announce the publication of the
<a class="reference external" href="http://dx.doi.org/10.1145/2743019">journal version of “Understanding Integer Overflow in C/C++”</a> <a class="footnote-reference" href="#id1" id="id2">[1]</a>, appearing
in <a class="reference external" href="http://tosem.acm.org/archive.cfm?id=2852270"><span class="caps">TOSEM</span> Volume 25 Issue 1</a> <a class="footnote-reference" href="#id4" id="id5">[2]</a>.
This is an updated and expanded version of our <a class="reference external" href="http://www.cs.utah.edu/~regehr/papers/overflow12.pdf"><span class="caps">ICSE12</span> paper</a> <a class="footnote-reference" href="#id6" id="id7">[3]</a> of the same name.
The longer journal format enabled a more thorough treatment of the subject, and
we did our best to take advantage of that opportunity.</p>
<p>Thanks to my co-authors for all their efforts, and especially for seeing this
work through to the end. It’s been a long run and you guys are great.</p>
<p></p>
<div class="section" id="highlights">
<h2>Highlights</h2>
<ul class="simple">
<li>Automated large-scale study of overflows in top 10,000 Debian packages (§6)</li>
<li>More thorough discussion and explanation of integer behavior, including implementation-defined behavior and usual arithmetic conversions (§§ 2, 3.1, 3.2)</li>
<li>Implementing recoverable checks efficiently: experience and two new optimizations (§4.4)</li>
<li>Deployment experiences and resulting improvements useful for anyone making compiler-based tools for the real world (§5).</li>
</ul>
<p>The Debian experiment was particularly fun and had many interesting results.
Full results are available on request, and we provide a <a class="reference external" href="http://wdtz.org/files/ioc-debian.log">complete version of selected results presented in Table <span class="caps">VI</span></a> <a class="footnote-reference" href="#id8" id="id9">[4]</a> online.</p>
</div>
<div class="section" id="available-now">
<h2>Available Now</h2>
<p>Paper is available <a class="reference external" href="http://dx.doi.org/10.1145/2743019">online</a> <a class="footnote-reference" href="#id1" id="id3">[1]</a> (via the <span class="caps">ACM</span> digital library) for your reading pleasure.</p>
<p>Let me know if you’d like to discuss our findings or any part of the paper. Enjoy!</p>
</div>
<hr class="docutils"/>
<div class="section" id="references">
<h2>References</h2>
<div class="footnote-table">
<table class="footnote" id="id1"><tr>
<td class="label">[1]</td>
<td class="fn-backrefs">↩<span class="fn-backref-list"> (<a class="fn-backref" href="#id2">1</a>, <a class="fn-backref" href="#id3">2</a>)</span></td>
<td><a class="reference external" href="http://dx.doi.org/10.1145/2743019">http://dx.doi.org/10.1145/2743019</a></td>
</tr></table>
<table class="footnote" id="id4"><tr>
<td class="label">[2]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id5">↩</a></td>
<td><a class="reference external" href="http://tosem.acm.org/archive.cfm?id=2852270">http://tosem.acm.org/archive.cfm?id=2852270</a></td>
</tr></table>
<table class="footnote" id="id6"><tr>
<td class="label">[3]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id7">↩</a></td>
<td><a class="reference external" href="http://www.cs.utah.edu/~regehr/papers/overflow12.pdf">http://www.cs.utah.edu/~regehr/papers/overflow12.pdf</a></td>
</tr></table>
<table class="footnote" id="id8"><tr>
<td class="label">[4]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id9">↩</a></td>
<td><a class="reference external" href="http://wdtz.org/files/ioc-debian.log">http://wdtz.org/files/ioc-debian.log</a></td>
</tr></table>
</div></div>
Freenode SASL Upgrade: Irssi HOWTO2015-01-05T12:25:00-06:002015-01-05T12:25:00-06:00Will Dietztag:wdtz.org,2015-01-05:/freenode-sasl-upgrade-irssi-howto.html<p>The <a class="reference external" href="http://freenode.net/">freenode</a> <span class="caps">IRC</span> network has for a long time supported
connecting and automatic identification using <a class="reference external" href="https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer"><span class="caps">SASL</span></a>.</p>
<p>Recently, the freenode network deprecated the commonly used
<span class="caps">SASL</span> mechanism <code class="docutils literal"><span class="pre"><span class="caps">DH</span>-<span class="caps">BLOWFISH</span></span></code> due to security concerns,
causing my <span class="caps">IRC</span> client (<a class="reference external" href="http://irssi.org/">irssi</a>) to no longer be able to authenticate.</p>
<p>Unfortunately, while scripts and guides describing using
irssi with <code class="docutils literal"><span class="pre"><span class="caps">DH</span>-<span class="caps">BLOWFISH</span></span></code> are plentiful, it seems the steps
required to use the new preferred
<code class="docutils literal"><span class="pre"><span class="caps">ECDSA</span>-<span class="caps">NIST256P</span>-<span class="caps">CHALLENGE</span></span></code> method are not yet documented.</p>
<p>Read on for a step-by-step walk-through of configuring
irssi to use <span class="caps">SASL</span> with freenode in 2015.</p>
<p></p><p>The <a class="reference external" href="http://freenode.net/">freenode</a> <a class="footnote-reference" href="#id1" id="id2">[1]</a> <span class="caps">IRC</span> network has for a long time supported
connecting and automatic identification using <a class="reference external" href="https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer"><span class="caps">SASL</span></a> <a class="footnote-reference" href="#id3" id="id4">[2]</a>.</p>
<p>Recently, the freenode network deprecated the commonly used
<span class="caps">SASL</span> mechanism <code class="docutils literal"><span class="pre"><span class="caps">DH</span>-<span class="caps">BLOWFISH</span></span></code> due to security concerns,
causing my <span class="caps">IRC</span> client (<a class="reference external" href="http://irssi.org/">irssi</a> <a class="footnote-reference" href="#id5" id="id6">[3]</a>) to no longer be able to authenticate.</p>
<p>Unfortunately, while scripts and guides describing using
irssi with <code class="docutils literal"><span class="pre"><span class="caps">DH</span>-<span class="caps">BLOWFISH</span></span></code> are plentiful, it seems the steps
required to use the new preferred
<code class="docutils literal"><span class="pre"><span class="caps">ECDSA</span>-<span class="caps">NIST256P</span>-<span class="caps">CHALLENGE</span></span></code> method are not yet documented.</p>
<p>Read on for a step-by-step walk-through of configuring
irssi to use <span class="caps">SASL</span> with freenode in 2015.</p>
<p></p>
<div class="section" id="background">
<h2>Background</h2>
<p>Recently <a class="reference external" href="http://blog.freenode.net/2014/11/atheme-7-2-and-freenode/">freenode upgraded to Atheme 7.2</a> <a class="footnote-reference" href="#id7" id="id8">[4]</a>, and in the
process deprecated support for the <span class="caps">SASL</span> mechanism
<code class="docutils literal"><span class="pre"><span class="caps">DH</span>-<span class="caps">BLOWFISH</span></span></code>.
Atheme is the reference implementation of the current <span class="caps">IRC</span>
protocol, which <a class="reference external" href="http://ircv3.atheme.org/documentation/sasl-dh-blowfish">deprecates <span class="caps">DH</span>-<span class="caps">BLOWFISH</span> in IRCv3</a> <a class="footnote-reference" href="#id9" id="id10">[5]</a>.</p>
<p>There are good reasons for this change, but regardless it’s
been done and irssi needs some help accommodating this change.</p>
<div class="section" id="why-not-plain">
<h3>Why not <span class="caps">PLAIN</span>?</h3>
<p>It would be remiss not to mention that the simplest solution
to this problem is to use the <code class="docutils literal"><span class="caps">PLAIN</span></code> <span class="caps">SASL</span> method in
conjunction with <span class="caps">SSL</span>. Clients configured in this way will
work with the new services just like they have previously,
with similar security properties.</p>
<p>While it is not my goal to convince you <code class="docutils literal"><span class="caps">PLAIN</span></code> is
insufficient, there is benefit in using a <span class="caps">SASL</span> method other
than <code class="docutils literal"><span class="caps">PLAIN</span></code> in a defense-in-depth sort of way. Should
the <span class="caps">SSL</span> stream become compromised in some manner, <code class="docutils literal"><span class="caps">PLAIN</span></code>
would make obtaining a user’s password as easy as forcing a
reconnect, while the other mechanisms provide additional
layers of security.</p>
</div>
<div class="section" id="use-ssl">
<h3>Use <span class="caps">SSL</span>!</h3>
<p>Regardless of the <span class="caps">SASL</span> method being used, if you’re
bothering with any of this the first and most effective step
to securing your <span class="caps">IRC</span> connection is using <span class="caps">SSL</span>.
<span class="caps">SSL</span> is supported by virtually all <span class="caps">IRC</span> networks and requires
only trivial configuration in most clients.</p>
<p>Be sure your client validates the server’s certificate
properly (strict <span class="caps">SSL</span>) or your connection is trivially
vulnerable to <a class="reference external" href="https://en.wikipedia.org/wiki/Man-in-the-middle_attack"><span class="caps">MITM</span></a> <a class="footnote-reference" href="#id11" id="id12">[6]</a> attacks.</p>
</div>
<div class="section" id="why-sasl-in-addition-to-ssl">
<h3>Why <span class="caps">SASL</span> in addition to <span class="caps">SSL</span></h3>
<p>Common implementations give <span class="caps">SASL</span> users one benefit not
generally available to other users: with <span class="caps">SASL</span>, network
services recognize you before you even are active on the
network, which can be useful when making use of services
like a hostname cloak or automatically joining channels only
open to invited accounts.</p>
<p>As an aside, as far as I can tell client-side certificates
(like those used with <a class="reference external" href="https://freenode.net/certfp/">CertFP</a> <a class="footnote-reference" href="#id13" id="id14">[7]</a> identification) could be used
to provide similar benefits but this doesn’t seem to be done
on any network I’m familiar with.
One possible explanation is that since CertFP doesn’t work
with Tor (I believe?), implementation efforts focus on <span class="caps">SASL</span>
which is available to all users.</p>
<p>Other reasons include additional layers of security in terms
of protecting the account password, and policies such as
<a class="reference external" href="https://freenode.net/irc_servers.shtml#tor">freenode’s requirement of <span class="caps">SASL</span> when connecting over Tor</a> <a class="footnote-reference" href="#id15" id="id16">[8]</a>.</p>
</div>
</div>
<div class="section" id="configuring-irssi-to-use-ecdsa-nist256p-challenge">
<h2>Configuring Irssi to use <span class="caps">ECDSA</span>-NIST256p-<span class="caps">CHALLENGE</span></h2>
<div class="section" id="install-ecdsatool">
<h3>1) Install ecdsatool</h3>
<p>First, download and build a copy of <a class="reference external" href="https://github.com/atheme/ecdsatool">ecdsatool</a> <a class="footnote-reference" href="#id17" id="id18">[9]</a>.
This wasn’t available as a package for my server’s
distribution, so I built is as follows:</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> git clone https://github.com/atheme/ecdsatool.git
<span class="gp">$</span> <span class="nb">cd</span> ecdsatool
<span class="gp">$</span> ./autogen.sh
<span class="gp">$</span> ./configure --prefix<span class="o">=</span><span class="nv">$HOME</span>/local
<span class="gp">$</span> make -j
<span class="gp">$</span> make install
</pre></div>
<p>Standard build recipe, tweak as you see fit.</p>
<p>Afterwards, be sure the resulting <code class="docutils literal">ecdsatool</code> utility is
available on your shell’s <code class="docutils literal"><span class="caps">PATH</span></code> so the irssi script we
configure later will be able to find and use it.</p>
</div>
<div class="section" id="generate-key-pair">
<h3>2) Generate key pair</h3>
<p>Next, use <code class="docutils literal">ecdsatool</code> to generate a key pair for <span class="caps">SASL</span> use:</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> mkdir -p ~/.irssi/certs
<span class="gp">$</span> ecdsatool keygen ~/.irssi/certs/freenode.pem
</pre></div>
<p>I keep my <span class="caps">IRC</span>-related certificates in <code class="docutils literal"><span class="pre">~/.irssi/certs</span></code>,
personal preference.</p>
</div>
<div class="section" id="install-cap-sasl-script">
<h3>3) Install cap_sasl script</h3>
<p>Next, grab a copy of the <code class="docutils literal">cap_sasl.pl</code> script shipped
in the ecdsatool repository:</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> mkdir -p ~/.irssi/scripts
<span class="gp">$</span> wget https://raw.githubusercontent.com/atheme/ecdsatool/master/cap_sasl.pl -O ~/.irssi/scripts/cap_sasl.pl
</pre></div>
<p>Additionally you likely want to have the script loaded when
irssi starts:</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> mkdir -p ~/.irssi/scripts/autorun
<span class="gp">$</span> ln -s ../cap_sasl.pl ~/.irssi/scripts/autorun/
</pre></div>
</div>
<div class="section" id="configure-sasl-for-freenode">
<h3>4) Configure <span class="caps">SASL</span> for freenode</h3>
<p>From within irssi, use the <code class="docutils literal">/sasl set</code> command to indicate
what username and certificate to use for your <span class="caps">IRC</span> network:</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> irssi
<span class="go">...</span>
</pre></div>
<div class="highlight"><pre><span></span>/sasl set freenode username /full/path/to/freenode.pem ECDSA-NIST256P-CHALLENGE
</pre></div>
<p>Replacing <code class="docutils literal">freenode</code> with the network name your configured
in irssi, <code class="docutils literal">username</code> with your freenode account name, and
the path with a full path to the key pair generated earlier.</p>
<p>Afterwards, be sure to save this information for future use:</p>
<div class="highlight"><pre><span></span>/sasl save
</pre></div>
<p>The result should be an entry in <code class="docutils literal"><span class="pre">~/.irssi/sasl.auth</span></code> that looks something like this:</p>
<div class="highlight"><pre><span></span>freenode dtzWill /home/will/.irssi/certs/freenode.pem ECDSA-NIST256P-CHALLENGE
</pre></div>
</div>
<div class="section" id="register-public-key-with-nickserv">
<h3>5) Register Public Key with NickServ</h3>
<p>Almost there! Final step is to give NickServ the public key
portion of our key pair so it can recognize your client and
associate it with your account.</p>
<p>First, grab the <code class="docutils literal">pubkey</code> from the key pair:</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> ecdsatool pubkey ~/.irssi/certs/freenode.pem
</pre></div>
<p>Next, connect to freenode and identify yourself as you would usually.</p>
<p>Finally, tell NickServ about your public key:</p>
<div class="highlight"><pre><span></span>/msg nickserv set property pubkey ArRZ4XCwSFYhT7RH5Ms7dosJEm8OYLO3gWSSGQCsYOCk
</pre></div>
<p>Replacing the example public key with what was printed by <code class="docutils literal">ecdsatool</code> in the previous step.</p>
</div>
<div class="section" id="done-reconnect-and-test">
<h3>6) Done! Reconnect and Test</h3>
<p>At this point you have all the pieces required to use <span class="caps">SASL</span>
with the <code class="docutils literal"><span class="pre"><span class="caps">ECDSA</span>-<span class="caps">NIST256P</span>-<span class="caps">CHALLENGE</span></span></code> mechanism to connect
to freenode.
Disconnect from freenode and reconnect to try it out!</p>
<p>If successful, you should see something like this:</p>
<div class="highlight"><pre><span></span>14:50 -!- Irssi: CLICAP: supported by server: account-notify extended-join identify-msg multi-prefix sasl
14:50 -!- Irssi: CLICAP: requesting: multi-prefix sasl
14:50 -!- Irssi: CLICAP: now enabled: multi-prefix sasl
14:50 -!- will!will@unaffiliated/dtzwill dtzWill You are now logged in as dtzWill.
14:50 -!- Irssi: SASL authentication successful
</pre></div>
</div>
</div>
<div class="section" id="alternative-method-without-ecdsatool">
<h2>Alternative Method Without ecdsatool</h2>
<p>It appears that there is another solution that does not
require the use of an external tool like <code class="docutils literal">ecdsatool</code> by
using the <code class="docutils literal"><span class="pre">Crypt::<span class="caps">PK</span>::<span class="caps">ECC</span></span></code> perl module.</p>
<p>This script is available in the Atheme git repository:
<a class="reference external" href="https://raw.githubusercontent.com/atheme/atheme/master/contrib/cap_sasl.pl">cap_sasl.pl git</a> <a class="footnote-reference" href="#id19" id="id20">[10]</a>.
In addition to no longer requiring an external tool, the
script offers a <code class="docutils literal">keygen</code> command that should make setup easier.</p>
<p>I haven’t tried this script yet myself, as I didn’t discover
it until well after I completed the procedure described
above.
Additionally, the module is uses doesn’t seem to be
available as a package on any of my systems although it can
of course be obtained using <a class="reference external" href="http://www.cpan.org/">cpan</a> <a class="footnote-reference" href="#id21" id="id22">[11]</a>.</p>
<p>If you try this method and have success, please report back.</p>
<div class="section" id="update-march-03-2015">
<h3>(Update: March 03, 2015)</h3>
<p>Jesper Freesbug from the comments was kind enough to share
his experiences and provide a walkthrough of the setup
process when using this approach. I’ve featured this
comment below and recommend taking a look if you’re
interested in this solution.</p>
<p>In addition to the FreeBSD package he mentions, it seems
other systems also provide the required perl module as part
of a <code class="docutils literal">cryptx</code> package. For example, on Arch it’s
available as an <span class="caps">AUR</span> package named <code class="docutils literal"><span class="pre">perl-cryptx</span></code>.
Hopefully the module is made more universally available in
the future.</p>
</div>
</div>
<div class="section" id="closing-thoughts">
<h2>Closing Thoughts</h2>
<p>It seems the folks working on Atheme and freenode are hard
at work improving the services that are widely used in a
variety of communities.
While this post is motivated by a lack of documentation, the
procedure is simple and it has been mentioned in multiple
places time that they hope to both document this thoroughly
soon and to improve the workflow for users.
Huge thanks to those folks, and for offering all of this
work for free for users like myself to enjoy.</p>
<p>Additionally, all of this is arguably something an <span class="caps">IRC</span>
client should support natively or at least help facilitate.
This is how some folks feel and have opened an issue
on the <a class="reference external" href="https://github.com/irssi/irssi/issues/4">irssi github</a> <a class="footnote-reference" href="#id23" id="id24">[12]</a>.</p>
<p>Hope this helps, and let me know if you have any questions or issues. Enjoy!</p>
</div>
<div class="section" id="references">
<h2>References</h2>
<div class="footnote-table">
<table class="footnote" id="id1"><tr>
<td class="label">[1]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id2">↩</a></td>
<td><a class="reference external" href="http://freenode.net/">http://freenode.net/</a></td>
</tr></table>
<table class="footnote" id="id3"><tr>
<td class="label">[2]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id4">↩</a></td>
<td><a class="reference external" href="https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer">https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer</a></td>
</tr></table>
<table class="footnote" id="id5"><tr>
<td class="label">[3]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id6">↩</a></td>
<td><a class="reference external" href="http://irssi.org/">http://irssi.org/</a></td>
</tr></table>
<table class="footnote" id="id7"><tr>
<td class="label">[4]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id8">↩</a></td>
<td><a class="reference external" href="http://blog.freenode.net/2014/11/atheme-7-2-and-freenode/">http://blog.freenode.net/2014/11/atheme-7-2-and-freenode/</a></td>
</tr></table>
<table class="footnote" id="id9"><tr>
<td class="label">[5]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id10">↩</a></td>
<td><a class="reference external" href="http://ircv3.atheme.org/documentation/sasl-dh-blowfish">http://ircv3.atheme.org/documentation/sasl-dh-blowfish</a></td>
</tr></table>
<table class="footnote" id="id11"><tr>
<td class="label">[6]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id12">↩</a></td>
<td><a class="reference external" href="https://en.wikipedia.org/wiki/Man-in-the-middle_attack">https://en.wikipedia.org/wiki/Man-in-the-middle_attack</a></td>
</tr></table>
<table class="footnote" id="id13"><tr>
<td class="label">[7]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id14">↩</a></td>
<td><a class="reference external" href="https://freenode.net/certfp/">https://freenode.net/certfp/</a></td>
</tr></table>
<table class="footnote" id="id15"><tr>
<td class="label">[8]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id16">↩</a></td>
<td><a class="reference external" href="https://freenode.net/irc_servers.shtml#tor">https://freenode.net/irc_servers.shtml#tor</a></td>
</tr></table>
<table class="footnote" id="id17"><tr>
<td class="label">[9]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id18">↩</a></td>
<td><a class="reference external" href="https://github.com/atheme/ecdsatool">https://github.com/atheme/ecdsatool</a></td>
</tr></table>
<table class="footnote" id="id19"><tr>
<td class="label">[10]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id20">↩</a></td>
<td><a class="reference external" href="https://raw.githubusercontent.com/atheme/atheme/master/contrib/cap_sasl.pl">https://raw.githubusercontent.com/atheme/atheme/master/contrib/cap_sasl.pl</a></td>
</tr></table>
<table class="footnote" id="id21"><tr>
<td class="label">[11]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id22">↩</a></td>
<td><a class="reference external" href="http://www.cpan.org/">http://www.cpan.org/</a></td>
</tr></table>
<table class="footnote" id="id23"><tr>
<td class="label">[12]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id24">↩</a></td>
<td><a class="reference external" href="https://github.com/irssi/irssi/issues/4">https://github.com/irssi/irssi/issues/4</a></td>
</tr></table>
</div></div>
Mysterious Lag Spikes and Faulty Switches2014-01-23T12:06:00-06:002014-01-23T12:06:00-06:00Will Dietztag:wdtz.org,2014-01-23:/mysterious-lag-spikes-and-faulty-switches.html<p>My residential internet experience has always been poor, so
when I started observing bizarre network behavior a few
months back I attributed it to upstream problems and wrote
it off with a sigh. The issue persisted however,
so over the winter break I decided to sit down
and tackle it once and for all.</p>
<p>Tracking down the source of the issue was an interesting
adventure, and ultimately the problem was in the last
place I thought to check: a pair of <span class="caps">TEG</span>-S80g unmanaged
gigabit switches made by TRENDnet.</p>
<p><strong>Do not buy!</strong> :)</p>
<p>Details follow.</p>
<p></p><p>My residential internet experience has always been poor, so
when I started observing bizarre network behavior a few
months back I attributed it to upstream problems and wrote
it off with a sigh. The issue persisted however,
so over the winter break I decided to sit down
and tackle it once and for all.</p>
<p>Tracking down the source of the issue was an interesting
adventure, and ultimately the problem was in the last
place I thought to check: a pair of <span class="caps">TEG</span>-S80g unmanaged
gigabit switches made by TRENDnet.</p>
<p><strong>Do not buy!</strong> :)</p>
<p>Details follow.</p>
<p></p>
<div class="section" id="the-problem">
<h2>The Problem</h2>
<p>First, let’s start with the symptom: a particular
online game was experiencing very severe “lag spikes”
every 60 seconds or so where it would seem I was
temporarily disconnected from the server.
This would last a few seconds then go back to normal.</p>
<p>Usually this sort of issue has simple explanations:
local modem temporarily lost connection with <span class="caps">ISP</span>,
a backup job is saturating my link causing packet
loss, or there’s transient upstream issue with
either my <span class="caps">ISP</span> or the game server.</p>
<p>Unfortunately all the likely candidates were quickly ruled
out. Connectivity was never lost as I was able to video chat
over Skype continuously through these events and
the logs on my modem were clean. This made me all
the more curious.</p>
<p>An upstream issue was a potential cause, but if so it was
a rather bizarre one: others on same server did not
experience similar issues, and further investigation
showed during these periods my game client was not
sending packets to my router at all which is strange.</p>
<p>Here’s a graph of network activity measured by
my router during these periods:</p>
<div class="figure-wrapper"><div class="figure align-center">
<a href="/images/packet_loss/problem_graph.png"><img alt="network activity graph during lag spikes" src="/images/packet_loss/problem_graph.png"/></a>
<p class="caption">Network Activity Graph -
The various dips to zero correspond with issues in-game.</p>
</div></div>
<p>The graph above shows network activity sampled every 3 seconds
over a period of 4 minutes, and the drops to zero correspond
with in-game issues. This selection of traffic shows at
least 6 outages of around 3 seconds during the 4-minute period.</p>
<p>This issue persisted for months since it only
had an impact on my weekly game nights with
my brother and we were still mostly able
to play. However, over the recent holiday
break I finally caved and treated myself
to a day of diving into this issue
in an attempt to finally answer the question
that had been bugging me for months:</p>
<p><strong>What’s going on?</strong></p>
</div>
<div class="section" id="a-step-forward">
<h2>A Step Forward</h2>
<p>The first big breakthrough was the ability
to reproduce this problem without
using my console or the game in question.
This enabled easily testing from various
parts of my network and more importantly
ruled out a buggy game client or server.</p>
<p>While gathering all the information I could
about the network normally and during
these periods I discovered the fantastic
<a class="reference external" href="http://netalyzr.icsi.berkeley.edu">NetAlyzr</a> <a class="footnote-reference" href="#id1" id="id2">[1]</a> tool from Berkeley.
This tool is an automated Java program that tests
for a number of common network problems and
reports them back to you.</p>
<p>The two most interesting issues turned up
from the <a class="reference external" href="http://netalyzr.icsi.berkeley.edu/restore/id=36ea240d-2034-f68c2f8d-d5b8-4a3e-8161/">initial NetAlyzr results</a> <a class="footnote-reference" href="#id5" id="id6">[3]</a> indicated
I was suffering from the famous <a class="reference external" href="http://www.bufferbloat.net/">buffer bloat</a> <a class="footnote-reference" href="#id3" id="id4">[2]</a>
problem, and experiencing
<strong>bursts of packet loss</strong>.
There’s not much I can do to change the size of
buffers upstream, but packet loss sure sounds related!</p>
<p>But what exactly does it mean by packet loss?
How is this measured? Is this something to contact
my <span class="caps">ISP</span> about? Unfortunately I was unable
to find answers to these questions in the
documentation, and couldn’t figure it out
from the code since the tool isn’t open-source.</p>
<p>To the debug-mobile!</p>
</div>
<div class="section" id="bursts-of-packet-loss">
<h2><span class="dquo">“</span>Bursts of Packet Loss”?</h2>
<p>To understand what the NetAlyzr tool was doing,
I captured the network traffic it sent and
received using <a class="reference external" href="http://www.wireshark.org/">Wireshark</a> <a class="footnote-reference" href="#id7" id="id8">[4]</a>. Together with
the server/client logs linked from the NetAlyzr
results page, I determined the tool exchanges
<span class="caps">UDP</span> packets every 20ms while running other
tests. This traffic communicates with their
server on the same port as other tests
executed concurrently so it takes a little
analysis for each capture to identify
the port used by the local endpoint.</p>
<p>Having filtered out the packet-loss-testing
traffic I graphed the packets/second to
look for issues:</p>
<div class="figure-wrapper"><div class="figure align-center">
<a href="/images/packet_loss/old_switch.png"><img alt="packet loss during NetAlyzr run" src="/images/packet_loss/old_switch.png"/></a>
<p class="caption">Captured NetAlyzr activity - black shows
traffic by the packet loss test, red shows
overall traffic. Packet loss occurs
around 84s to 87s and is marked
with a blue circle.</p>
</div></div>
<p>There’s the packet loss (see the blue circle)!
But why is this happening?
Note the loss didn’t happen during an earlier
traffic burst that peaked to about 10x higher,
suggesting something more subtle than
“packets are dropped under high load”.</p>
<p>To debug this, I started testing from various
points in the network starting with connecting
straight to my modem, then directly
to the router, working back towards my desktop.</p>
<p>I was surprised to discover the only location
between my desktop and modem that suffered
packet loss was when testing through my switch!</p>
<p>I immediately replaced the cable from
the switch to my router, which had no
effect. Similarly testing using that cable
directly resolved the packet loss issue.
Additionally this occurred regardless
which of the ports on the switch were used.</p>
<p>Starting to suspect my switch was somehow
to blame, I tried the procedure from an identical
switch in the living room. Same results:
packet loss when testing through the switch,
none when testing using the cable leading
to it.</p>
<p>…What?</p>
<p>Unfortunately my attempts to reproduce
this packet loss using my own synthetic
tests all failed, using various streams of
<span class="caps">TCP</span> and <span class="caps">UDP</span> data with the nifty <a class="reference external" href="http://iperf.sourceforge.net/">iperf</a> <a class="footnote-reference" href="#id9" id="id10">[5]</a> tool.
I only saw packet loss under traffic loads
that saturated the link, which is of course
the expected behavior.</p>
<p>I confirmed the game performed
properly when I removed the switch
from the network topology, which
was both relieving and frustrating:
what kind of junk switch drops
packet streams under these
basic circumstances?</p>
</div>
<div class="section" id="replacement">
<h2>Replacement</h2>
<p>I ended up purchasing replacement
<a class="reference external" href="http://www.amazon.com/gp/product/B00BZABOTU/">switches from <span class="caps">TP</span>-<span class="caps">LINK</span></a> <a class="footnote-reference" href="#id11" id="id12">[6]</a> that
have completely resolved the issue.</p>
<p>My <a class="reference external" href="http://netalyzr.icsi.berkeley.edu/restore/id=36ea240d-8613-5cfd6a62-667e-4a24-b979/">new NetAlyzr results</a> <a class="footnote-reference" href="#id13" id="id14">[7]</a> no longer
indicate packet loss, and the game
finally works as it should. Now,
if only I was any good at it :D.</p>
<p>Graphing a capture of the new results
no longer shows the interrupted connection:</p>
<div class="figure-wrapper"><div class="figure align-center">
<a href="/images/packet_loss/new_switch.png"><img alt="No packet loss during NetAlyzr run" src="/images/packet_loss/new_switch.png"/></a>
<p class="caption">Captured NetAlyzr activity - black shows
traffic by the packet loss test, red shows
overall traffic. No packet loss
occurred using the replacement switch.</p>
</div></div>
<p>A quick search suggests I’m not the only
one experiencing packet loss issues with
TRENDnet hardware, but nothing particularly
conclusive. One <a class="reference external" href="http://www.amazon.com/review/R2WV1S555TK8PU/ref=cm_cr_rdp_perm?ie=UTF8&ASIN=B0044GJ516&linkCode=&nodeID=">reviewer</a> <a class="footnote-reference" href="#id15" id="id16">[8]</a> of the same
<a class="reference external" href="http://www.amazon.com/gp/product/B001QUA6RA">faulty switches I purchased</a> <a class="footnote-reference" href="#id17" id="id18">[9]</a> did seem to
experience the same problem, but despite this the
product has great reviews overall. While it’s
possible I happened to get two from the same bad
batch, I can’t help but wonder if this isn’t a
design flaw present in all of these switches.</p>
<p>I have yet to contact TRENDnet about
this issue, but will be attempting
to refund or return the faulty products.
We’ll see how that goes :).</p>
</div>
<div class="section" id="summary">
<h2>Summary</h2>
<p>I had previously thought of my unmanaged switches
as incapable of basic failures such as this,
and will more thoroughly research and test
my hardware in the future.</p>
<p>If anyone has any insights that might explain
this behavior, I’m interested and willing
to provide the various packet captures
upon request. In the meantime, I’ll
be contacting TRENDnet about the issue
and looking for a refund or similar.</p>
<p>While these are only two switches, given
the crowded nature of the desktop networking
hardware market I’m going to stay away
from TRENDnet in the future and suggest
others do the same. Nothing unduly emotional,
but life is too short to risk basic network
components failing in this manner.</p>
<p>Chasing down the issue was a blast, and
I’m glad I can finally play the game
without constantly losing connection to server
in 3+ second bursts. More importantly,
I solved the puzzle of the strange network
behavior. Unfortunately now I’ll need to find a
new explanation for in-game mistakes! :)</p>
</div>
<div class="section" id="references">
<h2>References</h2>
<div class="footnote-table">
<table class="footnote" id="id1"><tr>
<td class="label">[1]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id2">↩</a></td>
<td><a class="reference external" href="http://netalyzr.icsi.berkeley.edu">http://netalyzr.icsi.berkeley.edu</a></td>
</tr></table>
<table class="footnote" id="id3"><tr>
<td class="label">[2]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id4">↩</a></td>
<td><a class="reference external" href="http://www.bufferbloat.net/">http://www.bufferbloat.net/</a></td>
</tr></table>
<table class="footnote" id="id5"><tr>
<td class="label">[3]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id6">↩</a></td>
<td><a class="reference external" href="http://netalyzr.icsi.berkeley.edu/restore/id=36ea240d-2034-f68c2f8d-d5b8-4a3e-8161/">http://netalyzr.icsi.berkeley.edu/restore/id=36ea240d-2034-f68c2f8d-d5b8-4a3e-8161/</a></td>
</tr></table>
<table class="footnote" id="id7"><tr>
<td class="label">[4]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id8">↩</a></td>
<td><a class="reference external" href="http://www.wireshark.org/">http://www.wireshark.org/</a></td>
</tr></table>
<table class="footnote" id="id9"><tr>
<td class="label">[5]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id10">↩</a></td>
<td><a class="reference external" href="http://iperf.sourceforge.net/">http://iperf.sourceforge.net/</a></td>
</tr></table>
<table class="footnote" id="id11"><tr>
<td class="label">[6]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id12">↩</a></td>
<td><a class="reference external" href="http://www.amazon.com/gp/product/B00BZABOTU/">http://www.amazon.com/gp/product/<span class="caps">B00BZABOTU</span>/</a></td>
</tr></table>
<table class="footnote" id="id13"><tr>
<td class="label">[7]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id14">↩</a></td>
<td><a class="reference external" href="http://netalyzr.icsi.berkeley.edu/restore/id=36ea240d-8613-5cfd6a62-667e-4a24-b979/">http://netalyzr.icsi.berkeley.edu/restore/id=36ea240d-8613-5cfd6a62-667e-4a24-b979/</a></td>
</tr></table>
<table class="footnote" id="id15"><tr>
<td class="label">[8]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id16">↩</a></td>
<td><a class="reference external" href="http://www.amazon.com/review/R2WV1S555TK8PU/ref=cm_cr_rdp_perm?ie=UTF8&ASIN=B0044GJ516&linkCode=&nodeID=">http://www.amazon.com/review/<span class="caps">R2WV1S555TK8PU</span>/ref=cm_cr_rdp_perm?ie=<span class="caps">UTF8</span>&<span class="caps">ASIN</span>=<span class="caps">B0044GJ516</span>&linkCode=&nodeID=</a></td>
</tr></table>
<table class="footnote" id="id17"><tr>
<td class="label">[9]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id18">↩</a></td>
<td><a class="reference external" href="http://www.amazon.com/gp/product/B001QUA6RA">http://www.amazon.com/gp/product/<span class="caps">B001QUA6RA</span></a></td>
</tr></table>
</div></div>
Catching pointer overflow bugs2013-11-18T18:05:00-06:002013-11-18T18:05:00-06:00Will Dietztag:wdtz.org,2013-11-18:/catching-pointer-overflow-bugs.html<p>In all varieties of C/C++, pointer arithmetic is undefined
if it overflows. That is to say the following example:</p>
<div class="highlight"><pre><span></span><span class="kt">void</span> <span class="nf">invalid</span><span class="p">(</span><span class="kt">char</span> <span class="o">*</span><span class="n">p</span><span class="p">)</span> <span class="p">{</span>
<span class="kt">char</span> <span class="o">*</span><span class="n">q</span> <span class="o">=</span> <span class="n">p</span> <span class="o">+</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"%p</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">p</span> <span class="o">-</span> <span class="p">(</span><span class="kt">uintptr_t</span><span class="p">)</span><span class="n">q</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
<p>invokes undefined behavior as it causes the pointer value to
wraparound to the equivalent of <code class="docutils literal"><span class="pre">-sizeof(char)</span></code>, which is
<code class="docutils literal">0xffffffffffffffff</code> on my 64bit system.</p>
<p>Unlike integer overflows which can be dangerous or benign
regardless of intention (<a class="reference external" href="http://www.cs.utah.edu/~regehr/papers/overflow12.pdf"><span class="caps">ICSE12</span></a>), pointer overflows are very unlikely
to be intentional and may be the source of a more serious
bug resulting in incorrect behavior or program crashing.</p>
<p></p><p>In all varieties of C/C++, pointer arithmetic is undefined
if it overflows. That is to say the following example:</p>
<div class="highlight"><pre><span></span><span class="kt">void</span> <span class="nf">invalid</span><span class="p">(</span><span class="kt">char</span> <span class="o">*</span><span class="n">p</span><span class="p">)</span> <span class="p">{</span>
<span class="kt">char</span> <span class="o">*</span><span class="n">q</span> <span class="o">=</span> <span class="n">p</span> <span class="o">+</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"%p</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">p</span> <span class="o">-</span> <span class="p">(</span><span class="kt">uintptr_t</span><span class="p">)</span><span class="n">q</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
<p>invokes undefined behavior as it causes the pointer value to
wraparound to the equivalent of <code class="docutils literal"><span class="pre">-sizeof(char)</span></code>, which is
<code class="docutils literal">0xffffffffffffffff</code> on my 64bit system.</p>
<p>Unlike integer overflows which can be dangerous or benign
regardless of intention (<a class="reference external" href="http://www.cs.utah.edu/~regehr/papers/overflow12.pdf"><span class="caps">ICSE12</span></a> <a class="footnote-reference" href="#id1" id="id2">[1]</a>), pointer overflows are very unlikely
to be intentional and may be the source of a more serious
bug resulting in incorrect behavior or program crashing.</p>
<p></p>
<div class="section" id="coming-soon-to-a-clang-near-you">
<h2>Coming Soon To a Clang Near You</h2>
<p>To address this issue I’ve built an extension to Clang that
checks for pointer overflows (<code class="docutils literal"><span class="pre">-fsanitize=pointer-overflow</span></code>),
and demonstrate its utility by using it to find bugs in a
variety of popular open source applications.</p>
<p>This will soon be available in mainline Clang as an addition
to the <code class="docutils literal"><span class="pre">-fsanitize=undefined</span></code> set of checks, helping reduce
dangerous occurrences of pointer overflow in the wild.</p>
<p>In the remainder of this post I describe the dangers
of pointer overflow, why existing tools are not
sufficient, and report results from running this tool
on a number of open source applications.</p>
</div>
<div class="section" id="why-pointer-overflow-is-dangerous">
<h2>Why Pointer Overflow is Dangerous</h2>
<p>Overflowed pointers can be dangerous in a variety of ways.
The most obvious is that an attempt to deference an
overflowed pointer will likely fail. These sorts of bugs
are already detected by tools such as <a class="reference external" href="http://valgrind.org/">valgrind</a> <a class="footnote-reference" href="#id5" id="id6">[3]</a>, <a class="reference external" href="http://code.google.com/p/address-sanitizer/">asan</a> <a class="footnote-reference" href="#id7" id="id8">[4]</a>, or
<a class="reference external" href="http://safecode.cs.illinois.edu/">SAFECode</a> <a class="footnote-reference" href="#id9" id="id10">[5]</a>. However these tools are inadequate for vetting
code against pointer overflow in the following ways.</p>
<div class="section" id="optimizations-assume-no-undefined-behavior">
<h3>Optimizations Assume No Undefined Behavior</h3>
<p>Today’s compilers are increasingly using optimizations that
assume code does not invoke undefined behavior. While such
optimizations can be useful (inferring loop bounds, removing
impossible conditions, assertion inference), they can also
cause unexpected and possibly dangerous behavior. An
example of this is the
<a class="reference external" href="https://wdtz.org/undefined-behavior-in-binutils-causes-segfault.html">undefined behavior bug in the latest binutils</a>
I’ve written about previously. By assuming pointer overflow
cannot occur, the code crashes at runtime due to a removed
“impossible” conditional expression. Similar optimizations
are performed by many compilers: a table of which compilers
perform which undefined behavior optimizations can be found
in a recent <a class="reference external" href="http://pdos.csail.mit.edu/papers/stack:sosp13.pdf"><span class="caps">SOSP</span>‘13 paper</a> <a class="footnote-reference" href="#id3" id="id4">[2]</a> by Xi Wang, et al (recommended
reading). These optimizations make the existence of
undefined behavior dangerous beyond the direct impact
of the value computed by the undefined expression.</p>
</div>
<div class="section" id="unintended-behavior-other-than-crashing">
<h3>Unintended Behavior Other Than Crashing</h3>
<p>Overflowed pointers can result in a variety of undesired
behavior that don’t involve dereferencing the result. For
example, an overflowed pointer might be used to compute an
incorrect offset or used in a comparison that causes
unintended behavior of the program.</p>
<p>Additionally, it’s common for data structures (such as
<span class="caps">LLVM</span>’s DenseMap) to reserve some pointer values
(<code class="docutils literal"><span class="pre">-1</span></code> and <code class="docutils literal"><span class="pre">-2</span></code>) as special values that are invalid to
use as keys. Through pointer overflow that generates these
values, the data structure might return the wrong value,
corrupt its data, or crash the program.</p>
</div>
<div class="section" id="early-detection-of-vulnerabilities">
<h3>Early Detection of Vulnerabilities</h3>
<p>Finally, early detection of pointer overflow can help
fix a potential vulnerability without needing to craft
a full exploit sufficient to trigger a memory safety
or security policy violation.</p>
<p>Together, these make the ability to check index expressions
dynamically for overflow an important part of the testing
process that’s not well met by today’s tools.</p>
</div>
</div>
<div class="section" id="new-sanitizer">
<h2>New Sanitizer</h2>
<p>In order to determine how common pointer overflow
is in the wild, I’ve extended Clang to optionally
add instrumentation to check for pointer overflow
and report any occurrences at runtime.</p>
<p>Clang already has support for adding similar runtime checks
for undefined behavior as part of <code class="docutils literal"><span class="pre">-fsanitize=undefined</span></code>,
the spiritual successor of our <a class="reference external" href="https://wdtz.org/pages/integer-overflow-checker-ioc.html"><span class="caps">IOC</span> (Integer Overflow
Checker)</a> project. The new sanitizer is
called <code class="docutils literal"><span class="pre">pointer-overflow</span></code>, and will be enabled as part of
<code class="docutils literal"><span class="pre">-fsanitize=undefined</span></code> once these features are accepted
upstream, bringing these important checks to the numerous
users already making use of <code class="docutils literal"><span class="pre">-fsanitize=undefined</span></code>.</p>
<p>The extension is straightforward: it hooks the various
places Clang generates <a class="reference external" href="http://llvm.org/docs/GetElementPtr.html"><span class="caps">GEP</span></a> <a class="footnote-reference" href="#id11" id="id12">[6]</a> instructions to add
additional code that converts the pointer to an integer
and performs checked arithmetic equivalent to the original
indexing expression. If the check fails, a call to the
sanitizer runtime is made that reports the error
to the user with a diagnostic similar to the following:</p>
<p><font color="white"><b>./test.c:7:19: <font color="red">runtime error:</font> pointer index expression with base 0x7fffffffd3cb overflowed to 0xffffffffffffffff</b></font></p><p>Indicating clearly to the user where in the source
the error occurred, as well as providing relevant
diagnostic information to assist in understanding
what happened so it may be fixed.</p>
</div>
<div class="section" id="pointer-overflows-in-the-wild">
<h2>Pointer Overflows in the Wild</h2>
<p>To motivate the addition of pointer overflow
checks to Clang, and to justify their inclusion
in <code class="docutils literal"><span class="pre">-fsanitize=undefined</span></code>, I built a variety
of open-source software with pointer overflow
checks enabled and ran their test-suites.
I report some of the bugs found below.</p>
<div class="section" id="self-host-testing-llvm-and-clang">
<h3>Self-Host: Testing <span class="caps">LLVM</span> and Clang</h3>
<p>A common practice in compilers is to use
your compiler to build itself, and ensure
the result still works. As part of testing
the robustness of the pointer overflow sanitizer
I did this, and was surprised to find that while
<span class="caps">LLVM</span> did not overflow any pointers, I did
find a bug in Clang’s <a class="reference external" href="http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20131028/091816.html">ASTVector</a> <a class="footnote-reference" href="#id13" id="id14">[7]</a> data structure.</p>
<p>The overflow occurred when attempting to insert
nothing to the end of an empty vector (simplified slightly):</p>
<div class="highlight"><pre><span></span><span class="n">iterator</span> <span class="nf">insert</span><span class="p">(</span><span class="n">iterator</span> <span class="n">pos</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">num</span><span class="p">,</span> <span class="k">const</span> <span class="n">T</span> <span class="o">&</span><span class="n">Elt</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">pos</span> <span class="o">==</span> <span class="k">this</span><span class="o">-></span><span class="n">end</span><span class="p">())</span> <span class="p">{</span>
<span class="n">append</span><span class="p">(</span><span class="n">num</span><span class="p">,</span> <span class="n">Elt</span><span class="p">);</span>
<span class="k">return</span> <span class="k">this</span><span class="o">-></span><span class="n">end</span><span class="p">()</span><span class="o">-</span><span class="mi">1</span><span class="p">;</span> <span class="c1">// <-- OVERFLOW</span>
<span class="p">}</span>
<span class="c1">// ...</span>
<span class="p">}</span>
</pre></div>
<p>This occurred most often when attempting to insert the contents
of an empty range into the vector, and occurs regularly
while running Clang’s tests.</p>
</div>
<div class="section" id="pcre-8-33">
<h3><span class="caps">PCRE</span> 8.33</h3>
<p>The latest version of the Perl Compatible Regular Expression (<span class="caps">PCRE</span>) library
triggers a pointer overflow in the following code during execution of its test-suite:</p>
<div class="highlight"><pre><span></span><span class="k">static</span> <span class="kt">int</span>
<span class="nf">match_ref</span><span class="p">(</span><span class="kt">int</span> <span class="n">offset</span><span class="p">,</span> <span class="k">register</span> <span class="n">PCRE_PUCHAR</span> <span class="n">eptr</span><span class="p">,</span> <span class="kt">int</span> <span class="n">length</span><span class="p">,</span> <span class="n">match_data</span> <span class="o">*</span><span class="n">md</span><span class="p">,</span>
<span class="n">BOOL</span> <span class="n">caseless</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">PCRE_PUCHAR</span> <span class="n">eptr_start</span> <span class="o">=</span> <span class="n">eptr</span><span class="p">;</span>
<span class="k">register</span> <span class="n">PCRE_PUCHAR</span> <span class="n">p</span> <span class="o">=</span> <span class="n">md</span><span class="o">-></span><span class="n">start_subject</span> <span class="o">+</span> <span class="n">md</span><span class="o">-></span><span class="n">offset_vector</span><span class="p">[</span><span class="n">offset</span><span class="p">];</span>
</pre></div>
<p>During execution of the addition in the last line of code. Interestingly,
the <cite>length</cite> parameter is always negative when this expression overflows,
which results in the function to return before using the dangerous pointer.</p>
<p>While this does not appear to be dangerous currently, there is debug code between
this calculation and the length check that a future change might cause to
use the faulty pointer value, and inlined calls inlined calls to this function
could be broken by compiler optimizations that rely on the assumption that this
is well-defined.</p>
<p>Luckily, this overflow can be easily fixed by moving the later check on
<cite>length</cite> to the function entry, which is my suggested solution.</p>
</div>
<div class="section" id="curl-7-32">
<h3>curl 7.32</h3>
<p>This program also overflowed a pointer during execution of its tests,
in particular during Test 138. Here, a null pointer is decremented
causing the overflow as shown in this excerpt from <code class="docutils literal">ftp.c</code>:</p>
<div class="highlight"><pre><span></span><span class="kt">char</span> <span class="o">*</span><span class="n">bytes</span><span class="p">;</span>
<span class="n">bytes</span><span class="o">=</span><span class="n">strstr</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span> <span class="s">" bytes"</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="n">bytes</span><span class="o">--</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="p">}</span>
</pre></div>
<p>Which overflows when the string “bytes” is not found and <code class="docutils literal">strstr</code> returns
<code class="docutils literal"><span class="caps">NULL</span></code>. Because it’s invalid to decrement a null pointer, an optimizing
compiler could assume bytes must be non-null and unconditionally execute the
code within. While I don’t know of a compiler that will take advantage of this
as described, but there’s no reason to assume this will be true of next year’s compilers.</p>
</div>
<div class="section" id="ffmpeg-2-0-2">
<h3>FFmpeg 2.0.2</h3>
<p>There was one occurrence of pointer overflow in FFmpeg
while running an instrumented version with its own
<span class="caps">FATE</span> test suite:</p>
<p><font color="white"><b>libavcodec/mpegvideo.c:3010:47: <font color="red">runtime error:</font> pointer index expression with base 0x000000000000 overflowed to 0xfffffffffffffff0</b></font></p><p>I’ve not had a chance to fully investigate this yet, but in the past FFmpeg has taken
integer overflow reports seriously and a quick mailing list search suggests they
have interest in purging pointer overflows as well.</p>
</div>
<div class="section" id="php-5-5-5">
<h3>php 5.5.5</h3>
<p>This software contained multiple pointer overflows. Two of these are due to expressions
that are evaluated <em>before</em> performing checks that abort the function. These can be easily
resolved by moving the indexing expressions after the safety checks, and are at risk
for an optimizing compiler to break the code as-is.</p>
<p>The other two are in macros <code class="docutils literal">EX_TMP_VAR</code> and <code class="docutils literal">EX_TMP_VAR_NUM</code>, which are currently defined
as follows:</p>
<div class="highlight"><pre><span></span><span class="cp">#define EX_TMP_VAR(ex, n) ((temp_variable*)(((char*)(ex)) + ((int)(n))))</span>
<span class="cp">#define EX_TMP_VAR_NUM(ex, n) (EX_TMP_VAR(ex, 0) - (1 + (n)))</span>
</pre></div>
<p>Which are used to translate between variable index and variable offsets, which
are intentionally negative but unfortunately expressed as pointers instead
of integer values. These macros can be fixed inplace by replacing with the following
messy equivalents:</p>
<div class="highlight"><pre><span></span><span class="cp">#define EX_TMP_VAR(ex, n) ((temp_variable*)((zend_uintptr_t)(ex) + sizeof(char)*((int)n)))</span>
<span class="cp">#define EX_TMP_VAR_NUM(ex, n) ((temp_variable*)((zend_uintptr_t)EX_TMP_VAR(ex, 0) - sizeof(temp_variable)*(1 + (n))))</span>
</pre></div>
<p>Which still produces questionably negative pointers, but through casts instead of indexing which
avoids the undefined behavior. It’s likely better to replace these mechanisms altogether
with something cleaner.</p>
</div>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>Pointer overflow is a common and serious problem that is poorly addressed
by today’s tools. Soon Clang will have support for finding occurrences
of this class of undefined behavior, ready to be used to help
improve the quality of your code.</p>
<p>Enjoy, and happy bug hunting :).</p>
</div>
<div class="section" id="references">
<h2>References</h2>
<div class="footnote-table">
<table class="footnote" id="id1"><tr>
<td class="label">[1]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id2">↩</a></td>
<td><a class="reference external" href="http://www.cs.utah.edu/~regehr/papers/overflow12.pdf">http://www.cs.utah.edu/~regehr/papers/overflow12.pdf</a></td>
</tr></table>
<table class="footnote" id="id3"><tr>
<td class="label">[2]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id4">↩</a></td>
<td><a class="reference external" href="http://pdos.csail.mit.edu/papers/stack:sosp13.pdf">http://pdos.csail.mit.edu/papers/stack:sosp13.pdf</a></td>
</tr></table>
<table class="footnote" id="id5"><tr>
<td class="label">[3]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id6">↩</a></td>
<td><a class="reference external" href="http://valgrind.org/">http://valgrind.org/</a></td>
</tr></table>
<table class="footnote" id="id7"><tr>
<td class="label">[4]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id8">↩</a></td>
<td><a class="reference external" href="http://code.google.com/p/address-sanitizer/">http://code.google.com/p/address-sanitizer/</a></td>
</tr></table>
<table class="footnote" id="id9"><tr>
<td class="label">[5]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id10">↩</a></td>
<td><a class="reference external" href="http://safecode.cs.illinois.edu/">http://safecode.cs.illinois.edu/</a></td>
</tr></table>
<table class="footnote" id="id11"><tr>
<td class="label">[6]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id12">↩</a></td>
<td><a class="reference external" href="http://llvm.org/docs/GetElementPtr.html">http://llvm.org/docs/GetElementPtr.html</a></td>
</tr></table>
<table class="footnote" id="id13"><tr>
<td class="label">[7]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id14">↩</a></td>
<td><a class="reference external" href="http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20131028/091816.html">http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20131028/091816.html</a></td>
</tr></table>
</div></div>
Kernel hacking: My very own BSOD2013-09-24T15:58:00-05:002013-09-24T15:58:00-05:00Will Dietztag:wdtz.org,2013-09-24:/kernel-hacking-my-very-own-bsod.html<p>While working on hacking the FreeBSD kernel
for a research project, I of course have
crashed things many times. Hooray for <span class="caps">VM</span>’s.</p>
<p>However, this latest time I’ve managed to really
mess things up. Instead of some esoteric message
in the logs, the result was the following being
written to the console:</p>
<div class="figure-wrapper"><div class="figure align-center">
<a href="/images/kernel_wut.png"><img alt="nonsense kernel console output" src="/images/kernel_wut.png"/></a>
<p class="caption">Nonsense Kernel console output</p>
</div></div>
<p>Hopefully those of you who’ve hacked on kernels previously
will appreciate the “wut” moment you get when your screen
looks like this instead of acting as expected :).</p>
<p></p><p>While working on hacking the FreeBSD kernel
for a research project, I of course have
crashed things many times. Hooray for <span class="caps">VM</span>’s.</p>
<p>However, this latest time I’ve managed to really
mess things up. Instead of some esoteric message
in the logs, the result was the following being
written to the console:</p>
<div class="figure-wrapper"><div class="figure align-center">
<a href="/images/kernel_wut.png"><img alt="nonsense kernel console output" src="/images/kernel_wut.png"/></a>
<p class="caption">Nonsense Kernel console output</p>
</div></div>
<p>Hopefully those of you who’ve hacked on kernels previously
will appreciate the “wut” moment you get when your screen
looks like this instead of acting as expected :).</p>
<p></p>
Bootswatch Theme Selector2013-08-25T21:54:00-05:002013-08-25T21:54:00-05:00Will Dietztag:wdtz.org,2013-08-25:/bootswatch-theme-selector.html<p>Despite my passion for light-on-dark themes,
not everyone finds them as readable and enjoyable
as I do. To address these concerns I investigated
what it would take to add a drop-down to this website
to enable dynamic selection of themes by the user.</p>
<div class="figure-wrapper"><div class="figure align-center">
<a href="/images/bootswatch_theme_dropdown.png"><img alt="bootswatch theme dropdown" src="/images/bootswatch_theme_dropdown.png"/></a>
<p class="caption">Bootswatch Theme Dropdown</p>
</div></div>
<p>The relevant code is given, but for those
interested in a complete example take a look
at the <a class="reference external" href="https://github.com/dtzWill/wdtz">website github</a>.</p>
<p></p><p>Despite my passion for light-on-dark themes,
not everyone finds them as readable and enjoyable
as I do. To address these concerns I investigated
what it would take to add a drop-down to this website
to enable dynamic selection of themes by the user.</p>
<div class="figure-wrapper"><div class="figure align-center">
<a href="/images/bootswatch_theme_dropdown.png"><img alt="bootswatch theme dropdown" src="/images/bootswatch_theme_dropdown.png"/></a>
<p class="caption">Bootswatch Theme Dropdown</p>
</div></div>
<p>The relevant code is given, but for those
interested in a complete example take a look
at the <a class="reference external" href="https://github.com/dtzWill/wdtz">website github</a> <a class="footnote-reference" href="#id1" id="id2">[1]</a>.</p>
<p></p>
<div class="section" id="adding-the-dropdown">
<h2>Adding the Dropdown</h2>
<p>The drop-down is based on a <a class="reference external" href="http://stackoverflow.com/a/17541994">StackOverflow post</a> <a class="footnote-reference" href="#id3" id="id4">[2]</a>,
modified to select a Bootswatch theme dynamically.</p>
<p>First, the <span class="caps">HTML</span> component of the drop-down:</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">li</span> <span class="na">class</span><span class="o">=</span><span class="s">"dropdown"</span> <span class="na">id</span><span class="o">=</span><span class="s">"theme-dropdown"</span><span class="p">></span>
<span class="p"><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"#"</span> <span class="na">class</span><span class="o">=</span><span class="s">"dropdown-toggle"</span> <span class="na">data-toggle</span><span class="o">=</span><span class="s">"dropdown"</span><span class="p">><</span><span class="nt">i</span> <span class="na">class</span><span class="o">=</span><span class="s">"icon-cogs icon-large"</span><span class="p">></</span><span class="nt">i</span><span class="p">></span> Theme<span class="p"><</span><span class="nt">b</span> <span class="na">class</span><span class="o">=</span><span class="s">"caret"</span><span class="p">></</span><span class="nt">b</span><span class="p">></</span><span class="nt">a</span><span class="p">></span>
<span class="p"><</span><span class="nt">ul</span> <span class="na">class</span><span class="o">=</span><span class="s">"dropdown-menu"</span><span class="p">></span>
<span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"#"</span> <span class="na">class</span><span class="o">=</span><span class="s">"change-style-menu-item"</span> <span class="na">rel</span><span class="o">=</span><span class="s">"cyborg"</span><span class="p">><</span><span class="nt">i</span> <span class="na">class</span><span class="o">=</span><span class="s">"icon-fixed-width icon-pencil"</span><span class="p">></</span><span class="nt">i</span><span class="p">></span> Cyborg (Default)<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span>
<span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"#"</span> <span class="na">class</span><span class="o">=</span><span class="s">"change-style-menu-item"</span> <span class="na">rel</span><span class="o">=</span><span class="s">"cerulean"</span><span class="p">><</span><span class="nt">i</span> <span class="na">class</span><span class="o">=</span><span class="s">"icon-fixed-width icon-pencil"</span><span class="p">></</span><span class="nt">i</span><span class="p">></span> Cerulean<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span>
<span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"#"</span> <span class="na">class</span><span class="o">=</span><span class="s">"change-style-menu-item"</span> <span class="na">rel</span><span class="o">=</span><span class="s">"cosmo"</span><span class="p">><</span><span class="nt">i</span> <span class="na">class</span><span class="o">=</span><span class="s">"icon-fixed-width icon-pencil"</span><span class="p">></</span><span class="nt">i</span><span class="p">></span> Cosmo<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span>
<span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"#"</span> <span class="na">class</span><span class="o">=</span><span class="s">"change-style-menu-item"</span> <span class="na">rel</span><span class="o">=</span><span class="s">"flatly"</span><span class="p">><</span><span class="nt">i</span> <span class="na">class</span><span class="o">=</span><span class="s">"icon-fixed-width icon-pencil"</span><span class="p">></</span><span class="nt">i</span><span class="p">></span> Flatly<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span>
<span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"#"</span> <span class="na">class</span><span class="o">=</span><span class="s">"change-style-menu-item"</span> <span class="na">rel</span><span class="o">=</span><span class="s">"journal"</span><span class="p">><</span><span class="nt">i</span> <span class="na">class</span><span class="o">=</span><span class="s">"icon-fixed-width icon-pencil"</span><span class="p">></</span><span class="nt">i</span><span class="p">></span> Journal<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span>
<span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"#"</span> <span class="na">class</span><span class="o">=</span><span class="s">"change-style-menu-item"</span> <span class="na">rel</span><span class="o">=</span><span class="s">"readable"</span><span class="p">><</span><span class="nt">i</span> <span class="na">class</span><span class="o">=</span><span class="s">"icon-fixed-width icon-pencil"</span><span class="p">></</span><span class="nt">i</span><span class="p">></span> Readable<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span>
<span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"#"</span> <span class="na">class</span><span class="o">=</span><span class="s">"change-style-menu-item"</span> <span class="na">rel</span><span class="o">=</span><span class="s">"slate"</span><span class="p">><</span><span class="nt">i</span> <span class="na">class</span><span class="o">=</span><span class="s">"icon-fixed-width icon-pencil"</span><span class="p">></</span><span class="nt">i</span><span class="p">></span> Slate<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span>
<span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"#"</span> <span class="na">class</span><span class="o">=</span><span class="s">"change-style-menu-item"</span> <span class="na">rel</span><span class="o">=</span><span class="s">"spacelab"</span><span class="p">><</span><span class="nt">i</span> <span class="na">class</span><span class="o">=</span><span class="s">"icon-fixed-width icon-pencil"</span><span class="p">></</span><span class="nt">i</span><span class="p">></span> Spacelab<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span>
<span class="p"></</span><span class="nt">ul</span><span class="p">></span>
<span class="p"></</span><span class="nt">li</span><span class="p">></span>
</pre></div>
<p>Which as-written belongs in the navbar of your
Bootstrap-based site. Here I have drop-down items for my
preferred Bootswatch themes, but the format should
straightforward to add/remove as you see fit.</p>
<p>The important tidbits are the
<code class="docutils literal"><span class="pre">class="change-style-menu-item"</span></code> and <code class="docutils literal"><span class="pre">rel="spacelab"</span></code>
fields of the theme links, the rest is Bootstrap-specific
code for putting it all into a drop-down.</p>
<p>Additionally, we’re going to modify the primary bootswatch
theme link by adding a title to it for easy lookup from
javascript later:</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">link</span> <span class="na">href</span><span class="o">=</span><span class="s">"//netdna.bootstrapcdn.com/bootswatch/3.0.0/cyborg/bootstrap.min.css"</span> <span class="na">rel</span><span class="o">=</span><span class="s">"stylesheet"</span> <span class="na">title</span><span class="o">=</span><span class="s">"main"</span><span class="p">></span>
</pre></div>
<div class="section" id="javascript">
<h3>Javascript</h3>
<p>Now to perform the desired theme change,
add the following jquery function:</p>
<div class="highlight"><pre><span></span><span class="cm">/* When a theme-change item is selected, update theme */</span>
<span class="nx">jQuery</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">$</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">(</span><span class="s1">'body'</span><span class="p">).</span><span class="nx">on</span><span class="p">(</span><span class="s1">'click'</span><span class="p">,</span> <span class="s1">'.change-style-menu-item'</span><span class="p">,</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">theme_name</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="k">this</span><span class="p">).</span><span class="nx">attr</span><span class="p">(</span><span class="s1">'rel'</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">theme</span> <span class="o">=</span> <span class="s2">"//netdna.bootstrapcdn.com/bootswatch/3.0.0/"</span> <span class="o">+</span> <span class="nx">theme_name</span> <span class="o">+</span> <span class="s2">"/bootstrap.min.css"</span><span class="p">;</span>
<span class="nx">set_theme</span><span class="p">(</span><span class="nx">theme</span><span class="p">);</span>
<span class="p">});</span>
<span class="p">});</span>
</pre></div>
<p>What does this do? This adds a function to each of the
<code class="docutils literal"><span class="pre">change-style-menu-item</span></code> links we added in our <span class="caps">HTML</span>
earlier which is triggered when the link is clicked.</p>
<p>When invoked, this function extracts the value of the <code class="docutils literal">rel</code> attribute
of the clicked link, and uses it to invoke <code class="docutils literal">set_theme</code>
with an appropriate replacement <span class="caps">CSS</span> <span class="caps">URL</span>.</p>
<p>Below is a tentative definition for <code class="docutils literal">set_theme</code> that we’ll
be replacing in the following section.</p>
<div class="highlight"><pre><span></span><span class="kd">function</span> <span class="nx">set_theme</span><span class="p">(</span><span class="nx">theme</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">(</span><span class="s1">'link[title="main"]'</span><span class="p">).</span><span class="nx">attr</span><span class="p">(</span><span class="s1">'href'</span><span class="p">,</span> <span class="nx">theme</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
</div>
</div>
<div class="section" id="making-the-selection-persist">
<h2>Making The Selection Persist</h2>
<p>What we’ve done so far adds the drop-down and lets users
change the theme. That’s nifty and highlights the magic
of Bootswatch themes, but what if we wanted to make
the user’s selection persist across visits and
as they navigate the site?</p>
<p>To accomplish this I opted to use <span class="caps">HTML5</span>’s
<a class="reference external" href="http://diveintohtml5.info/storage.html">local storage</a> <a class="footnote-reference" href="#id5" id="id6">[3]</a> feature. Being a simple blog
without a concept of users, stashing this server-side
makes little sense, and local storage is supported
by all modern browsers and is very easy to use.</p>
<div class="section" id="saving-theme-with-local-storage">
<h3>Saving Theme with Local Storage</h3>
<p>First, we add a function to determine if the user supports
the local storage feature. This helps avoid errors on
browsers without support or with the feature disabled:</p>
<div class="highlight"><pre><span></span><span class="kd">function</span> <span class="nx">supports_html5_storage</span><span class="p">()</span> <span class="p">{</span>
<span class="k">try</span> <span class="p">{</span>
<span class="k">return</span> <span class="s1">'localStorage'</span> <span class="k">in</span> <span class="nb">window</span> <span class="o">&&</span> <span class="nb">window</span><span class="p">[</span><span class="s1">'localStorage'</span><span class="p">]</span> <span class="o">!==</span> <span class="kc">null</span><span class="p">;</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="kc">false</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="kd">var</span> <span class="nx">supports_storage</span> <span class="o">=</span> <span class="nx">supports_html5_storage</span><span class="p">();</span>
</pre></div>
<p>Next let’s replace our <code class="docutils literal">set_theme()</code> function with one that
saves the selected them into local storage:</p>
<div class="highlight"><pre><span></span><span class="kd">function</span> <span class="nx">set_theme</span><span class="p">(</span><span class="nx">theme</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">(</span><span class="s1">'link[title="main"]'</span><span class="p">).</span><span class="nx">attr</span><span class="p">(</span><span class="s1">'href'</span><span class="p">,</span> <span class="nx">theme</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">supports_storage</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">localStorage</span><span class="p">.</span><span class="nx">theme</span> <span class="o">=</span> <span class="nx">theme</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>Finally, add code to load the setting and apply it
if we find the user has a saved theme choice:</p>
<div class="highlight"><pre><span></span><span class="cm">/* On load, set theme from local storage */</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">supports_storage</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">theme</span> <span class="o">=</span> <span class="nx">localStorage</span><span class="p">.</span><span class="nx">theme</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">theme</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">set_theme</span><span class="p">(</span><span class="nx">theme</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="cm">/* Don't annoy user with options that don't persist */</span>
<span class="nx">$</span><span class="p">(</span><span class="s1">'#theme-dropdown'</span><span class="p">).</span><span class="nx">hide</span><span class="p">();</span>
<span class="p">}</span>
</pre></div>
<p>Where I chose to hide the drop-down altogether for
clients that don’t support local storage.</p>
</div>
</div>
<div class="section" id="selectively-enabling-on-development-builds">
<h2>Selectively Enabling on Development Builds</h2>
<p>Ultimately I decided to not publish this on the
production version of the website, only enabling
it in development builds. A website’s design
is an important part of capturing the author’s
voice and by giving control over this to visitors
the expressivity of the blog is weakened. Kudos
to my friend Brian for pointing this out.</p>
<p>In this section I describe the easy Pelican-specific changes
needed to only include the theme-selection code in
development but not in production builds.</p>
<p>First, I added the following new definition to <code class="docutils literal">pelicanconf.py</code>:</p>
<div class="highlight"><pre><span></span><span class="n">THEME_CHANGER</span> <span class="o">=</span> <span class="kc">True</span>
</pre></div>
<p>and the following to <code class="docutils literal">publishconf.py</code>:</p>
<div class="highlight"><pre><span></span><span class="n">THEME_CHANGER</span> <span class="o">=</span> <span class="kc">False</span>
</pre></div>
<p>Next, I wrapped the various <span class="caps">HTML</span> and javascript
components in <code class="docutils literal">base.html</code> with</p>
<div class="highlight"><pre><span></span><span class="cp">{%</span> <span class="k">if</span> <span class="nv">THEME_CHANGER</span> <span class="cp">%}</span><span class="x"></span>
<span class="x"> ...</span>
<span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span><span class="x"></span>
</pre></div>
<p>Which has an effect very much like C preprocessor <code class="docutils literal">#ifdef
THEME_CHANGER</code> … <code class="docutils literal">#endif</code>, only including the theme
switcher dropdown and supporting javascript when not using
the publish configuration.</p>
</div>
<hr class="docutils"/>
<div class="section" id="references">
<h2>References</h2>
<div class="footnote-table">
<table class="footnote" id="id1"><tr>
<td class="label">[1]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id2">↩</a></td>
<td><a class="reference external" href="https://github.com/dtzWill/wdtz">https://github.com/dtzWill/wdtz</a></td>
</tr></table>
<table class="footnote" id="id3"><tr>
<td class="label">[2]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id4">↩</a></td>
<td><a class="reference external" href="http://stackoverflow.com/a/17541994">http://stackoverflow.com/a/17541994</a></td>
</tr></table>
<table class="footnote" id="id5"><tr>
<td class="label">[3]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id6">↩</a></td>
<td><a class="reference external" href="http://diveintohtml5.info/storage.html">http://diveintohtml5.info/storage.html</a></td>
</tr></table>
</div></div>
Easy Website Tuning2013-08-25T20:12:00-05:002013-08-25T20:12:00-05:00Will Dietztag:wdtz.org,2013-08-25:/easy-website-tuning.html<p>Having recently moved to <a class="reference external" href="http://getpelican.com">Pelican</a>,
I found myself interested in seeing
how I could coerce my server (runing <a class="reference external" href="http://httpd.apache.org/">Apache</a>)
into taking proper advantage of the bulk
of the website being static.</p>
<p>There’s lots of information on the subject
out there, but below I describe the few
easy changes I made to greatly improve
performance of my website.</p>
<p></p><p>Having recently moved to <a class="reference external" href="http://getpelican.com">Pelican</a> <a class="footnote-reference" href="#id2" id="id3">[1]</a>,
I found myself interested in seeing
how I could coerce my server (runing <a class="reference external" href="http://httpd.apache.org/">Apache</a> <a class="footnote-reference" href="#id6" id="id7">[2]</a>)
into taking proper advantage of the bulk
of the website being static.</p>
<p>There’s lots of information on the subject
out there, but below I describe the few
easy changes I made to greatly improve
performance of my website.</p>
<p></p>
<div class="section" id="tunables">
<h2>Tunables</h2>
<div class="section" id="enable-gzip-compression">
<h3>Enable gzip compression</h3>
<p>Ensure you have <code class="docutils literal">mod_deflate</code> available and add
the following to your httpd.conf:</p>
<div class="highlight"><pre><span></span><span class="nt"><IfModule</span> <span class="s">mod_deflate.c</span><span class="nt">></span>
<span class="nb">AddOutputFilterByType</span> DEFLATE text/text text/html text/plain text/xml text/css application/x-javascript application/javascript application/json
<span class="nt"></IfModule></span>
</pre></div>
<p>Which enables gzip compression on the listed file
types, which is simple and appropriate for a
<a class="reference external" href="http://getpelican.com">Pelican</a> <a class="footnote-reference" href="#id2" id="id4">[1]</a>-based website.</p>
<p>While trimming some of my larger dependencies
(such as the bootstrap js and css) would be
useful, gzip compression is simple and very
effective on text-based formats such as these.</p>
<p>Since compression support is used automatically in
response to the client’s reported support for it,
I was happy to note that <code class="docutils literal">mod_deflate</code>
automaticaly adds an appropriate <code class="docutils literal">Vary:
<span class="pre">Accept-Encoding</span></code> header to enable correct
functionality even when accessed through a proxy.</p>
</div>
<div class="section" id="set-the-cache-control-header">
<h3>Set the “Cache-Control” header</h3>
<p>This header is used to specify the
caching behavior of the client,
and when used properly can greatly
reduce accesses made to unchanging
files while browsing your site.</p>
<p>To add these headers, one can make
use of <code class="docutils literal">mod_expires</code> or by simply
adding something like the following:</p>
<div class="highlight"><pre><span></span><span class="nt"><FilesMatch</span> <span class="s">"\.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf)$"</span><span class="nt">></span>
<span class="nb">Header</span> set Cache-Control <span class="s2">"max-age=29030400, public"</span>
<span class="nt"></FilesMatch></span>
</pre></div>
<p>This example says the content is
<em>public</em>, which allows any intermediate host
to cache the contents. For static resources
like those matched by this directive,
this is reasonable behavior. You would
not want this to be set for content
generated for a particular user.</p>
<p>The <code class="docutils literal"><span class="pre">max-age</span></code> component specifies how
long (in seconds) the content can be cached.</p>
</div>
</div>
<div class="section" id="use-a-cdn">
<h2>Use a <span class="caps">CDN</span></h2>
<p>From Wikipedia, a <a class="reference external" href="https://en.wikipedia.org/wiki/Content_delivery_network"><span class="caps">CDN</span></a> <a class="footnote-reference" href="#id8" id="id9">[3]</a> is:</p>
<blockquote>
A content delivery network or content distribution network
(<span class="caps">CDN</span>) is a large distributed system of servers deployed in
multiple data centers across the Internet. The goal of a <span class="caps">CDN</span>
is to serve content to end-users with high availability and
high performance.</blockquote>
<p>In short, <span class="caps">CDN</span>’s are used to provide high-speed
access to a file by providing the data from a
server “close” to the user accessing it.</p>
<div class="section" id="outsourcing-dependency-hosting">
<h3>Outsourcing Dependency Hosting</h3>
<p>While the content of my blog is small (text mostly), it does have
some moderately hefty dependencies:</p>
<ul class="simple">
<li><a class="reference external" href="http://getbootstrap.com/">Bootstrap</a> <a class="footnote-reference" href="#id10" id="id11">[4]</a>: Provides primary design elements used by this site</li>
<li><a class="reference external" href="http://jquery.com/">jquery</a> <a class="footnote-reference" href="#id12" id="id13">[5]</a>: Bootstrap dependency</li>
<li><a class="reference external" href="http://fortawesome.github.io/Font-Awesome/">Font Awesome</a> <a class="footnote-reference" href="#id14" id="id15">[6]</a>: Scalable icons without need for images</li>
<li><a class="reference external" href="http://bootswatch.com/">Bootswatch</a> <a class="footnote-reference" href="#id16" id="id17">[7]</a>: For theming bootstrap</li>
</ul>
<p>Luckily there are a number of free <span class="caps">CDN</span>’s that host very
common files such as these. These are maintained by large
companies with the explicit purpose of being highly
available and very fast, enabling users accessing my site to
get copies of these dependencies much faster than I’d be
able to provide them otherwise.</p>
<p>The <span class="caps">CDN</span>’s I’m now using are:</p>
<ul class="simple">
<li><a class="reference external" href="http://www.bootstrapcdn.com/">Bootstrap <span class="caps">CDN</span></a> <a class="footnote-reference" href="#id18" id="id19">[8]</a>: Used to host my Bootstrap, Font-Awesome, and theme dependencies.</li>
<li><a class="reference external" href="https://developers.google.com/speed/libraries/devguide?hl=ja#Libraries">Google Hosted Libraries</a> <a class="footnote-reference" href="#id20" id="id21">[9]</a>: Hosts jquery</li>
</ul>
</div>
<div class="section" id="cdn-example">
<h3><span class="caps">CDN</span> Example</h3>
<p>An example of rewriting some self-hosted dependencies to use
<span class="caps">CDN</span>-hosted sources is shown below:</p>
<p>Before:</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">link</span> <span class="na">rel</span><span class="o">=</span><span class="s">"stylesheet"</span> <span class="na">href</span><span class="o">=</span><span class="s">"{{ SITEURL }}/theme/css/bootstrap.min.css"</span> <span class="na">type</span><span class="o">=</span><span class="s">"text/css"</span> <span class="p">/></span>
</pre></div>
<p>After:</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">link</span> <span class="na">href</span><span class="o">=</span><span class="s">"//netdna.bootstrapcdn.com/bootstrap/2.3.2/css/bootstrap.min.css"</span> <span class="na">rel</span><span class="o">=</span><span class="s">"stylesheet"</span><span class="p">></span>
</pre></div>
<p>Where <code class="docutils literal">{{ <span class="caps">SITEURL</span> }}</code> is part of the templating done by
<a class="reference external" href="http://getpelican.com">Pelican</a> <a class="footnote-reference" href="#id2" id="id5">[1]</a>, and the <code class="docutils literal">//</code> prefix to the <span class="caps">CDN</span> source is a trick
to use http or https dependencing on how the current page is loaded.</p>
</div>
</div>
<div class="section" id="tools-to-spot-easy-tunables">
<h2>Tools to Spot Easy Tunables</h2>
<p>While investigating how to improve my website’s loading times,
I ran into three tools that were particularly useful.</p>
<div class="section" id="gtmetrix">
<h3>GTMetrix</h3>
<p><a class="reference external" href="http://gtmetrix.com/">GTMetrix</a> <a class="footnote-reference" href="#id24" id="id25">[11]</a></p>
<p>Providing a page load waterfall graph, YSlow/PageSpeed scores with detailed explanations,
this was my favorite as the most comprehensive of these tools.</p>
</div>
<div class="section" id="pingdom">
<h3>Pingdom</h3>
<p><a class="reference external" href="http://tools.pingdom.com/fpt/">Pingdom Website Speed Test</a> <a class="footnote-reference" href="#id22" id="id23">[10]</a></p>
<p>Also offers a waterfall, and another “Page Speed” analysis that found
other problems not reported by GTMetrix.</p>
</div>
<div class="section" id="google">
<h3>Google</h3>
<p><a class="reference external" href="https://developers.google.com/speed/pagespeed/insights/">Google PageSpeed Insights</a> <a class="footnote-reference" href="#id26" id="id27">[12]</a></p>
<p>Google’s offering based on its <a class="reference external" href="https://developers.google.com/speed/pagespeed/">PageSpeed</a> <a class="footnote-reference" href="#id28" id="id29">[13]</a> service, this was particularly helpful
in identifying issues for mobile users. They also offer plugins that run
on your server and automatically optimize your website. While very cool
and appealing, I didn’t explore this route since I was interested
in the learning to be had by exploring this by hand.</p>
</div>
</div>
<div class="section" id="summary">
<h2>Summary</h2>
<p>This blog now loads quite a bit faster, and ignoring time happily spent
reading up on the various topics discussed here the actual
changes were straightforward and easy. Hope this helps!</p>
</div>
<hr class="docutils"/>
<div class="section" id="references">
<h2>References</h2>
<div class="footnote-table">
<table class="footnote" id="id2"><tr>
<td class="label">[1]</td>
<td class="fn-backrefs">↩<span class="fn-backref-list"> (<a class="fn-backref" href="#id3">1</a>, <a class="fn-backref" href="#id4">2</a>, <a class="fn-backref" href="#id5">3</a>)</span></td>
<td><a class="reference external" href="http://getpelican.com">http://getpelican.com</a></td>
</tr></table>
<table class="footnote" id="id6"><tr>
<td class="label">[2]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id7">↩</a></td>
<td><a class="reference external" href="http://httpd.apache.org/">http://httpd.apache.org/</a></td>
</tr></table>
<table class="footnote" id="id8"><tr>
<td class="label">[3]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id9">↩</a></td>
<td><a class="reference external" href="https://en.wikipedia.org/wiki/Content_delivery_network">https://en.wikipedia.org/wiki/Content_delivery_network</a></td>
</tr></table>
<table class="footnote" id="id10"><tr>
<td class="label">[4]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id11">↩</a></td>
<td><a class="reference external" href="http://getbootstrap.com/">http://getbootstrap.com/</a></td>
</tr></table>
<table class="footnote" id="id12"><tr>
<td class="label">[5]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id13">↩</a></td>
<td><a class="reference external" href="http://jquery.com/">http://jquery.com/</a></td>
</tr></table>
<table class="footnote" id="id14"><tr>
<td class="label">[6]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id15">↩</a></td>
<td><a class="reference external" href="http://fortawesome.github.io/Font-Awesome/">http://fortawesome.github.io/Font-Awesome/</a></td>
</tr></table>
<table class="footnote" id="id16"><tr>
<td class="label">[7]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id17">↩</a></td>
<td><a class="reference external" href="http://bootswatch.com/">http://bootswatch.com/</a></td>
</tr></table>
<table class="footnote" id="id18"><tr>
<td class="label">[8]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id19">↩</a></td>
<td><a class="reference external" href="http://www.bootstrapcdn.com/">http://www.bootstrapcdn.com/</a></td>
</tr></table>
<table class="footnote" id="id20"><tr>
<td class="label">[9]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id21">↩</a></td>
<td><a class="reference external" href="https://developers.google.com/speed/libraries/devguide?hl=ja#Libraries">https://developers.google.com/speed/libraries/devguide?hl=ja#Libraries</a></td>
</tr></table>
<table class="footnote" id="id22"><tr>
<td class="label">[10]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id23">↩</a></td>
<td><a class="reference external" href="http://tools.pingdom.com/fpt/">http://tools.pingdom.com/fpt/</a></td>
</tr></table>
<table class="footnote" id="id24"><tr>
<td class="label">[11]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id25">↩</a></td>
<td><a class="reference external" href="http://gtmetrix.com/">http://gtmetrix.com/</a></td>
</tr></table>
<table class="footnote" id="id26"><tr>
<td class="label">[12]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id27">↩</a></td>
<td><a class="reference external" href="https://developers.google.com/speed/pagespeed/insights/">https://developers.google.com/speed/pagespeed/insights/</a></td>
</tr></table>
<table class="footnote" id="id28"><tr>
<td class="label">[13]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id29">↩</a></td>
<td><a class="reference external" href="https://developers.google.com/speed/pagespeed/">https://developers.google.com/speed/pagespeed/</a></td>
</tr></table>
</div></div>
Bye, Bye, Buick2013-08-19T15:12:00-05:002013-08-19T15:12:00-05:00Will Dietztag:wdtz.org,2013-08-19:/bye-bye-buick.html<p>Today finally got rid of my old ‘89 Buick Century, sold off for parts.</p>
<p>Memorial post to my first and well-loved car follows.</p>
<p></p><p>Today finally got rid of my old ‘89 Buick Century, sold off for parts.</p>
<p>Memorial post to my first and well-loved car follows.</p>
<p></p>
<p>Here’s to you, ol’ faithful. Pictures taken right before it
got hauled off to the scrapyard in the sky. Err, down the street.</p>
<p><a class="reference external" href="images/buick/5.jpg"><img alt="Driver-side of Buick" src="images/buick/5-thumb.jpg"/></a>
<a class="reference external" href="/images/buick/6.jpg"><img alt="Front of Buick" src="/images/buick/6-thumb.jpg"/></a>
<a class="reference external" href="/images/buick/3.jpg"><img alt="Passenger-side of Buick" src="/images/buick/3-thumb.jpg"/></a></p>
<p>We’ve been through a lot, old friend! You’ll be missed!</p>
<div class="section" id="the-beginning">
<h2>The Beginning</h2>
<p>Buick was born in Springfield, and was lucky enough
to attend <span class="caps">WUSTL</span> in its youth before settling down
with good folk in Chambana. A conservative car,
it only had two owners in its life.</p>
<p>I met Buick when I was looking to sell my old <span class="caps">TV</span>
and my friend was looking to find a good home
for it. Seeing a mutually beneficial opportunity,
a trade was arranged.</p>
<p>It’s amusing to notice it all started with
a car and <span class="caps">TV</span> upgrade: this time I’m finding myself
with a new car and passing on the Buick, and friend
I got it from just upgraded to a new <span class="caps">TV</span>.</p>
</div>
<div class="section" id="oh-the-places-you-ve-gone">
<h2>Oh the Places You’ve Gone</h2>
<ul class="simple">
<li>Many trips to <span class="caps">WI</span></li>
<li>Many trips down to southern <span class="caps">IL</span></li>
</ul>
<p>This past summer, we gave the Buick one last
hurrah and drove down to St. Louis for a weekend
vacation, even stopping off at its birthplace on
the return trip. As always, it served us well.</p>
</div>
<div class="section" id="the-last-days">
<h2>The Last Days</h2>
<p>Other than a minor alternator replacement a few
years ago, Buick always hid its age well.
However, recently it started to show signs
that its time has come:</p>
<ul class="simple">
<li>Driver-side door doesn’t close properly</li>
<li>Repair shop borked R12-><span class="caps">R134A</span> upgrade, killing A/C</li>
<li>Blinker unit broken, apparently non-trivial to fix (??)</li>
<li>Large piece of undercarriage fell off in <span class="caps">WI</span> somewhere</li>
<li>Rear-view mirror fell off</li>
<li>Original hubcaps stolen, attempted to steal car (lots of interior damage)</li>
<li>Gas mileage plummeted</li>
<li>Rust creeping everywhere</li>
</ul>
<p>So the decision was made to move on and let Buick
rest in peace (pieces?). Also, the safety of
driving it was rather questionable at this point…</p>
</div>
<div class="section" id="a-new-chapter">
<h2>A New Chapter</h2>
<p>Replacing the old car is one of many changes I’ve
been making lately, an easy example being this website.</p>
<p>That said, the old adage about old friends being
gold comes to mind and indeed there are good
things about the previous chapter that will be missed.</p>
<p>As they say, your first love^H^H^H^Hcar will always
have a special place in your heart… :)</p>
<p>So with that, I say:</p>
<blockquote class="epigraph">
So long, and thanks for all the fish!</blockquote>
<p><a class="reference external" href="/images/buick/1.jpg"><img alt="Buick Alternator" src="/images/buick/1-thumb.jpg"/></a>
<a class="reference external" href="/images/buick/2.jpg"><img alt="Buick Engine" src="/images/buick/2-thumb.jpg"/></a></p>
</div>
Undefined Behavior in Binutils Causes Segfault2013-08-19T14:24:00-05:002013-08-19T14:24:00-05:00Will Dietztag:wdtz.org,2013-08-19:/undefined-behavior-in-binutils-causes-segfault.html<p>As reported on the <a class="reference external" href="http://sourceware.org/bugzilla/show_bug.cgi?id=15836">binutils bugzilla</a>.</p>
<p>No response yet, but should be easy to fix.</p>
<p>Details follow (same as in bug report but easier to read).</p>
<p></p><p>As reported on the <a class="reference external" href="http://sourceware.org/bugzilla/show_bug.cgi?id=15836">binutils bugzilla</a> <a class="footnote-reference" href="#id2" id="id3">[1]</a>.</p>
<p>No response yet, but should be easy to fix.</p>
<p>Details follow (same as in bug report but easier to read).</p>
<p></p>
<div class="section" id="description">
<h2>Description</h2>
<p>The file <code class="docutils literal"><span class="pre">tc-i386-intel.c</span></code> contains undefined behavior in
recent binutils versions (including latest at time of writing).</p>
<p>The error occurs in multiple places
(for example <a class="reference external" href="http://sourceware.org/git/?p=binutils.git;a=blob;f=gas/config/tc-i386-intel.c;h=3f6b057613451839c796ca8a9cdbef2fe6532ec6;hb=HEAD#l432">line 432</a> <a class="footnote-reference" href="#id4" id="id5">[2]</a>, reproduced below) and is incorrect
as it assumes unsigned integer wrapping semantics for
pointer arithmetic on the variable <code class="docutils literal">scale</code> in a number of
places. In particular, the check:</p>
<div class="highlight"><pre><span></span><span class="mi">432</span> <span class="k">if</span> <span class="p">(</span><span class="n">ret</span> <span class="o">&&</span> <span class="n">scale</span> <span class="o">&&</span> <span class="p">(</span><span class="n">scale</span> <span class="o">+</span> <span class="mi">1</span><span class="p">))</span>
</pre></div>
<p>Gets optimized to <code class="docutils literal">if (ret && scale)</code> because it is impossible for <code class="docutils literal">scale + 1</code> to evaluate to <code class="docutils literal"><span class="caps">NULL</span></code> without invoking undefined behavior. Note that the earlier decrement from <code class="docutils literal"><span class="caps">NULL</span></code> is also invalid, and possibly other constructs in related code.</p>
<p>This is is a problem as it results in the conditional being taken when scale is <code class="docutils literal"><span class="pre">(int*)-sizeof(int)</span></code>, which leads to an invalid pointer being dereferenced in <code class="docutils literal">resolve_expression()</code>.</p>
</div>
<div class="section" id="steps-to-reproduce">
<h2>Steps to reproduce</h2>
<ol class="arabic simple">
<li>Obtain and unpack binutils 2.22 or latest via git (tested with 0b0b7b5).</li>
<li>Obtain clang 3.3 or latest trunk (from your package manager or build) and modify <span class="caps">PATH</span> as appropriate.</li>
<li>Configure similar to the following:</li>
</ol>
<div class="highlight"><pre><span></span><span class="gp">$</span> <span class="nv">CC</span><span class="o">=</span>clang <span class="nv">CXX</span><span class="o">=</span>clang++ ./configure --disable-werror --enable-ld<span class="o">=</span>no
</pre></div>
<ol class="arabic simple" start="4">
<li>Build.</li>
</ol>
<div class="highlight"><pre><span></span><span class="gp">$</span> make -j
</pre></div>
<ol class="arabic simple" start="5">
<li>Run the just-built ‘as’ using the following program from the testsuite:</li>
</ol>
<div class="highlight"><pre><span></span><span class="gp">$</span> valgrind gas/as-new --32 gas/testsuite/gas/i386/intelbad.s
</pre></div>
<ol class="arabic simple" start="6">
<li>Observe segfault, see referenced <a class="reference internal" href="#id1">valgrind.log</a> for the output of the above command.</li>
</ol>
</div>
<div class="section" id="impact">
<h2>Impact</h2>
<p>Presently prevents building a functional binutils with recent versions of
clang, and is a time-bomb for breaking future builds. Compilers (including gcc
and clang) are known to increasingly take advantage of undefined behavior in
newer versions and so this may be an issue in the future even with
compilers/platforms that safely build this today.</p>
</div>
<div class="section" id="valgrind-log">
<h2>valgrind.log</h2>
<p>Valgrind log from step #5 above:</p>
<pre class="literal-block" id="id1">
==80032== Memcheck, a memory error detector
==80032== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==80032== Using Valgrind-3.8.0.SVN and LibVEX; rerun with -h for copyright info
==80032== Command: gas/as-new --32 gas/testsuite/gas/i386/intelbad.s
==80032==
...
==80032== Invalid read of size 8
==80032== at 0x40D881: resolve_expression (expr.c:2026)
==80032== by 0x42EA9F: i386_intel_simplify (tc-i386-intel.c:415)
==80032== by 0x42E8A2: i386_intel_simplify (tc-i386-intel.c:297)
==80032== by 0x42E5F7: i386_intel_simplify (tc-i386-intel.c:297)
==80032== by 0x427835: md_assemble (tc-i386-intel.c:537)
==80032== by 0x415E1E: read_a_source_file (read.c:950)
==80032== by 0x404606: main (as.c:1089)
==80032== Address 0xffffffffffffffe0 is not stack'd, malloc'd or (recently) free'd
==80032==
==80032==
==80032== Process terminating with default action of signal 11 (SIGSEGV)
==80032== Access not within mapped region at address 0xFFFFFFFFFFFFFFE0
==80032== at 0x40D881: resolve_expression (expr.c:2026)
==80032== by 0x42EA9F: i386_intel_simplify (tc-i386-intel.c:415)
==80032== by 0x42E8A2: i386_intel_simplify (tc-i386-intel.c:297)
==80032== by 0x42E5F7: i386_intel_simplify (tc-i386-intel.c:297)
==80032== by 0x427835: md_assemble (tc-i386-intel.c:537)
==80032== by 0x415E1E: read_a_source_file (read.c:950)
==80032== by 0x404606: main (as.c:1089)
==80032== If you believe this happened as a result of a stack
==80032== overflow in your program's main thread (unlikely but
==80032== possible), you can try to increase the size of the
==80032== main thread stack using the --main-stacksize= flag.
==80032== The main thread stack size used in this run was 10485760.
==80032==
==80032== HEAP SUMMARY:
==80032== in use at exit: 3,931,677 bytes in 1,740 blocks
==80032== total heap usage: 1,894 allocs, 154 frees, 3,963,363 bytes allocated
==80032==
==80032== LEAK SUMMARY:
==80032== definitely lost: 0 bytes in 0 blocks
==80032== indirectly lost: 0 bytes in 0 blocks
==80032== possibly lost: 0 bytes in 0 blocks
==80032== still reachable: 3,931,677 bytes in 1,740 blocks
==80032== suppressed: 0 bytes in 0 blocks
==80032== Rerun with --leak-check=full to see details of leaked memory
==80032==
==80032== For counts of detected and suppressed errors, rerun with: -v
==80032== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 4 from 4)
</pre>
</div>
<hr class="docutils"/>
<div class="section" id="references">
<h2>References</h2>
<div class="footnote-table">
<table class="footnote" id="id2"><tr>
<td class="label">[1]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id3">↩</a></td>
<td><a class="reference external" href="http://sourceware.org/bugzilla/show_bug.cgi?id=15836">http://sourceware.org/bugzilla/show_bug.cgi?id=15836</a></td>
</tr></table>
<table class="footnote" id="id4"><tr>
<td class="label">[2]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id5">↩</a></td>
<td><a class="reference external" href="http://sourceware.org/git/?p=binutils.git;a=blob;f=gas/config/tc-i386-intel.c;h=3f6b057613451839c796ca8a9cdbef2fe6532ec6;hb=HEAD#l432">http://sourceware.org/git/?p=binutils.git;a=blob;f=gas/config/tc-i386-intel.c;h=3f6b057613451839c796ca8a9cdbef2fe6532ec6;hb=<span class="caps">HEAD</span>#l432</a></td>
</tr></table>
</div></div>
Analysis of Integer Error in Latest wget2013-08-19T14:03:00-05:002013-08-19T14:03:00-05:00Will Dietztag:wdtz.org,2013-08-19:/analysis-of-integer-error-in-latest-wget.html<p>Here’s another integer error found by our research
that occurs in wget 1.14 (latest at time of writing)
in the <code class="docutils literal"><span class="pre">--version</span></code> output.</p>
<p>This has been reported and fixed upstream,
coming soon to a wget near you!</p>
<p></p><p>Here’s another integer error found by our research
that occurs in wget 1.14 (latest at time of writing)
in the <code class="docutils literal"><span class="pre">--version</span></code> output.</p>
<p>This has been reported and fixed upstream,
coming soon to a wget near you!</p>
<p></p>
<div class="section" id="description">
<h2>Description</h2>
<p>The error is a signedness comparison issue
in <code class="docutils literal">format_and_print_line()</code> used by wget’s
<code class="docutils literal"><span class="pre">--version</span></code> to print information such as
the flags used to the compiler and linker
when wget was built.</p>
<p>Relevant code from src/main.c:</p>
<div class="highlight"><pre><span></span><span class="kt">int</span> <span class="n">remaining_chars</span><span class="p">;</span>
<span class="p">...</span>
<span class="n">token</span> <span class="o">=</span> <span class="n">strtok</span> <span class="p">(</span><span class="n">line_dup</span><span class="p">,</span> <span class="s">" "</span><span class="p">);</span>
<span class="k">while</span> <span class="p">(</span><span class="n">token</span> <span class="o">!=</span> <span class="nb">NULL</span><span class="p">)</span>
<span class="p">{</span>
<span class="cm">/* If however a token is much larger than the maximum</span>
<span class="cm"> line length, all bets are off and we simply print the</span>
<span class="cm"> token on the next line. */</span>
<span class="k">if</span> <span class="p">(</span><span class="n">remaining_chars</span> <span class="o"><=</span> <span class="n">strlen</span> <span class="p">(</span><span class="n">token</span><span class="p">))</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">printf</span> <span class="p">(</span><span class="s">"</span><span class="se">\n</span><span class="s">%*c"</span><span class="p">,</span> <span class="n">TABULATION</span><span class="p">,</span> <span class="sc">' '</span><span class="p">)</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span>
<span class="k">return</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>
<span class="n">remaining_chars</span> <span class="o">=</span> <span class="n">line_length</span> <span class="o">-</span> <span class="n">TABULATION</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="n">printf</span> <span class="p">(</span><span class="s">"%s "</span><span class="p">,</span> <span class="n">token</span><span class="p">)</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span>
<span class="k">return</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>
<span class="n">remaining_chars</span> <span class="o">-=</span> <span class="n">strlen</span> <span class="p">(</span><span class="n">token</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">;</span> <span class="cm">/* account for " " */</span>
<span class="n">token</span> <span class="o">=</span> <span class="n">strtok</span> <span class="p">(</span><span class="nb">NULL</span><span class="p">,</span> <span class="s">" "</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
<p>Where if <code class="docutils literal">remaining_chars</code> goes negative the comparison
<code class="docutils literal">remaining_chars <= strlen (token)</code> erroneously returns
true, causing all remaining tokens to be printed on the same
line instead of being wrapped.</p>
<p>When the printed string contains a token longer than
the wrapping width (<code class="docutils literal">line_length - <span class="caps">TABULATION</span></code> in the above)
<code class="docutils literal">remaining_chars</code> will go negative and trigger this issue. Humorously we encountered this issue only because of a very long flag used by our research compiler that took a path name as an argument.</p>
</div>
<div class="section" id="status">
<h2>Status</h2>
<p>This issue has been <a class="reference external" href="https://savannah.gnu.org/bugs/index.php?39453">reported</a> <a class="footnote-reference" href="#id1" id="id2">[1]</a>, and is now fixed <a class="reference external" href="http://git.savannah.gnu.org/cgit/wget.git/commit/?id=a12bd59111bd5e6fba91a8f1fa6c09698d03f740">upstream</a> <a class="footnote-reference" href="#id3" id="id4">[2]</a>
thanks to the wget developers working with me on the issue.
While touching that code, also prettified wget’s
<code class="docutils literal"><span class="pre">--version`</span></code> output in general (see below), and scored
my first (minor) entry on a <span class="caps">GNU</span> tool’s ChangeLog. Woo!</p>
</div>
<div class="section" id="example">
<h2>Example</h2>
<p>Before:</p>
<pre class="literal-block">
GNU Wget 1.14.74-8bf9-dirty built on linux-gnu.
+digest +https +ipv6 +iri +large-file +nls +ntlm +opie +ssl/gnutls
Wgetrc:
/usr/local/etc/wgetrc (system)
Locale: /usr/local/share/locale
Compile: gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
-DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I../lib
-DUNIMPORTANT_TEXT_TO_CREATE_VERY_LONG_TOKEN_IN_FLAG_STRING=123456789 -O2 -g
Link: gcc
-DUNIMPORTANT_TEXT_TO_CREATE_VERY_LONG_TOKEN_IN_FLAG_STRING=123456789 -O2 -g -lnettle -lgnutls -lgcrypt -lgpg-error -lz -lz -lidn -luuid -lpcre ftp-opie.o gnutls.o http-ntlm.o ../lib/libgnu.a
</pre>
<p>After:</p>
<pre class="literal-block">
GNU Wget 1.14.74-8bf9-dirty built on linux-gnu.
+digest +https +ipv6 +iri +large-file +nls +ntlm +opie +ssl/gnutls
Wgetrc:
/usr/local/etc/wgetrc (system)
Locale:
/usr/local/share/locale
Compile:
gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
-DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I../lib
-DUNIMPORTANT_TEXT_TO_CREATE_VERY_LONG_TOKEN_IN_FLAG_STRING=123456789
-O2 -g
Link:
gcc
-DUNIMPORTANT_TEXT_TO_CREATE_VERY_LONG_TOKEN_IN_FLAG_STRING=123456789
-O2 -g -lnettle -lgnutls -lgcrypt -lgpg-error -lz -lz -lidn -luuid
-lpcre ftp-opie.o gnutls.o http-ntlm.o ../lib/libgnu.a
</pre>
</div>
<hr class="docutils"/>
<div class="section" id="references">
<h2>References:</h2>
<div class="footnote-table">
<table class="footnote" id="id1"><tr>
<td class="label">[1]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id2">↩</a></td>
<td><a class="reference external" href="https://savannah.gnu.org/bugs/index.php?39453">https://savannah.gnu.org/bugs/index.php?39453</a></td>
</tr></table>
<table class="footnote" id="id3"><tr>
<td class="label">[2]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id4">↩</a></td>
<td><a class="reference external" href="http://git.savannah.gnu.org/cgit/wget.git/commit/?id=a12bd59111bd5e6fba91a8f1fa6c09698d03f740">http://git.savannah.gnu.org/cgit/wget.git/commit/?id=a12bd59111bd5e6fba91a8f1fa6c09698d03f740</a></td>
</tr></table>
</div></div>
Several curl Globbing Bugs Reported, Fixed2013-08-19T10:00:00-05:002013-08-19T10:00:00-05:00Will Dietztag:wdtz.org,2013-08-19:/several-curl-globbing-bugs-reported-fixed.html<p>I recently discovered a slew of bugs in <a class="reference external" href="http://curl.haxx.se/">curl</a>,
which occur in all versions I have access to,
including latest release at time of writing (7.32.0)
and goes back to at least 7.19.7.</p>
<p>One of the bugs can be used to crash curl
or systems using curl via exec (not libcurl),
the others cause strange or incorrect behavior.</p>
<p></p><p>I recently discovered a slew of bugs in <a class="reference external" href="http://curl.haxx.se/">curl</a> <a class="footnote-reference" href="#id1" id="id2">[1]</a>,
which occur in all versions I have access to,
including latest release at time of writing (7.32.0)
and goes back to at least 7.19.7.</p>
<p>One of the bugs can be used to crash curl
or systems using curl via exec (not libcurl),
the others cause strange or incorrect behavior.</p>
<p></p>
<p>The bugs in question are:</p>
<ul class="simple">
<li><a class="reference external" href="https://sourceforge.net/p/curl/bugs/1264">crasher</a> <a class="footnote-reference" href="#id3" id="id4">[2]</a> due to bad error handling</li>
<li><a class="reference external" href="https://sourceforge.net/p/curl/bugs/1265">sscanf</a> <a class="footnote-reference" href="#id5" id="id6">[3]</a>-based parsing overflow</li>
<li><a class="reference external" href="https://sourceforge.net/p/curl/bugs/1266">input validation</a> <a class="footnote-reference" href="#id7" id="id8">[4]</a> bug on the ”step” portion of a range glob</li>
<li><a class="reference external" href="https://sourceforge.net/p/curl/bugs/1267"><span class="caps">URL</span> count overflow</a> <a class="footnote-reference" href="#id9" id="id10">[5]</a> bug triggered by globbing for a ridiculous number of <span class="caps">URL</span>’s</li>
</ul>
<p>These have all been fixed now, first as part of a general
<a class="reference external" href="https://github.com/bagder/curl/commit/5ca96cb84410270e233c92bf1b2583cba40c3fad">globbing overhaul</a> <a class="footnote-reference" href="#id11" id="id12">[6]</a> commit, followed by a specific
<a class="reference external" href="https://github.com/bagder/curl/commit/f15a88f2b25ee44d8c8d3bdcf2508fdf50f8b868"><span class="caps">URL</span> overflow checking</a> <a class="footnote-reference" href="#id13" id="id14">[7]</a> fix.</p>
<p>See the bug reports for details on the errors and example
invocations, hopefully a release fixing these issues is made soon.</p>
<p>These errors were encountered during my research on integer
overflows, and I’m glad the developer fixed them so quickly!
Hopefully these fixes will reach everyone in the form of a
new release soon :).</p>
<hr class="docutils"/>
<div class="section" id="references">
<h2>References</h2>
<div class="footnote-table">
<table class="footnote" id="id1"><tr>
<td class="label">[1]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id2">↩</a></td>
<td><a class="reference external" href="http://curl.haxx.se/">http://curl.haxx.se/</a></td>
</tr></table>
<table class="footnote" id="id3"><tr>
<td class="label">[2]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id4">↩</a></td>
<td><a class="reference external" href="https://sourceforge.net/p/curl/bugs/1264">https://sourceforge.net/p/curl/bugs/1264</a></td>
</tr></table>
<table class="footnote" id="id5"><tr>
<td class="label">[3]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id6">↩</a></td>
<td><a class="reference external" href="https://sourceforge.net/p/curl/bugs/1265">https://sourceforge.net/p/curl/bugs/1265</a></td>
</tr></table>
<table class="footnote" id="id7"><tr>
<td class="label">[4]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id8">↩</a></td>
<td><a class="reference external" href="https://sourceforge.net/p/curl/bugs/1266">https://sourceforge.net/p/curl/bugs/1266</a></td>
</tr></table>
<table class="footnote" id="id9"><tr>
<td class="label">[5]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id10">↩</a></td>
<td><a class="reference external" href="https://sourceforge.net/p/curl/bugs/1267">https://sourceforge.net/p/curl/bugs/1267</a></td>
</tr></table>
<table class="footnote" id="id11"><tr>
<td class="label">[6]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id12">↩</a></td>
<td><a class="reference external" href="https://github.com/bagder/curl/commit/5ca96cb84410270e233c92bf1b2583cba40c3fad">https://github.com/bagder/curl/commit/5ca96cb84410270e233c92bf1b2583cba40c3fad</a></td>
</tr></table>
<table class="footnote" id="id13"><tr>
<td class="label">[7]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id14">↩</a></td>
<td><a class="reference external" href="https://github.com/bagder/curl/commit/f15a88f2b25ee44d8c8d3bdcf2508fdf50f8b868">https://github.com/bagder/curl/commit/f15a88f2b25ee44d8c8d3bdcf2508fdf50f8b868</a></td>
</tr></table>
</div></div>
New Website2013-08-19T09:00:00-05:002013-08-19T09:00:00-05:00Will Dietztag:wdtz.org,2013-08-19:/new-website.html<p>Obligatory post announcing new website:
changes, lessons, and a chance
to give the new site a spin.</p>
<p>Maybe I’ll stick with the whole blog
thing this time around :).</p>
<p></p><p>Obligatory post announcing new website:
changes, lessons, and a chance
to give the new site a spin.</p>
<p>Maybe I’ll stick with the whole blog
thing this time around :).</p>
<p></p>
<div class="section" id="why-pelican">
<h2>Why Pelican</h2>
<p>I finally replaced the old <a class="reference external" href="http://wordpress.com">Wordpress</a> <a class="footnote-reference" href="#id1" id="id2">[1]</a>-based website with one
based on <a class="reference external" href="http://getpelican.com">Pelican</a> <a class="footnote-reference" href="#id3" id="id4">[2]</a>. Pelican is a static site generator which
is appealing to me for the following reasons:</p>
<ul class="simple">
<li>Content creation can be done from the comfort of vim</li>
<li><span class="caps">CLI</span> goodness for deployment</li>
<li>git goodness for easy backup, content management, and writing on-the-go</li>
<li>super-easy to host, allowing me to drop php and mysql support on my server.</li>
</ul>
<p>… which is basically what is listed on the Pelican website. Good work.</p>
<p>First few reasons just make me happy, which is especially
important if I’m going to succeed at actually posting periodically.</p>
<p>The last reason is technical, but important because the server running
this site is on its last legs (and was reject hardware years ago).
As an added bonus I no longer have to worry about a wordpress
vulnerability giving my site to some hacker. Being able to
trivially deploy to an alternate host should my server go
down is also a nice comfort.</p>
<p>Finally, Pelican is used for <a class="reference external" href="http://kernel.org">kernel.org</a> <a class="footnote-reference" href="#id6" id="id7">[3]</a>. Good enough for them,
good enough for me.</p>
</div>
<div class="section" id="external">
<h2>External</h2>
<p>For the curious, here’s the components used to build this website:</p>
<p>Git repository: <a class="reference external" href="https://github.com/dtzWill/wdtz">github</a> <a class="footnote-reference" href="#id8" id="id9">[4]</a></p>
<p>Pelican theme used: <a class="reference external" href="https://github.com/masterkoppa/Pelican-Themes/tree/master/bootstraped">bootstraped</a> <a class="footnote-reference" href="#id10" id="id11">[5]</a></p>
<p>Bootstrap coloring: <a class="reference external" href="http://bootswatch.com/cyborg/">Cyborg</a> <a class="footnote-reference" href="#id12" id="id13">[6]</a></p>
</div>
<div class="section" id="lessons">
<h2>Lessons</h2>
<ul class="simple">
<li>Never underestimate how much time can be lost redesigning a website.</li>
<li>Nothing makes you go “didn’t need those files anyway” like a runaway script</li>
<li><a class="reference external" href="http://getpelican.com">Pelican</a> <a class="footnote-reference" href="#id3" id="id5">[2]</a> is awesome</li>
<li>I believe I now understand the appeal of formats like
<a class="reference external" href="http://docutils.sourceforge.net/rst.html">rst</a> <a class="footnote-reference" href="#id14" id="id15">[7]</a>:
More writing, less formatting.</li>
<li>Writing (even this post!) is remarkably hard. Practice
makes perfect.</li>
</ul>
</div>
<hr class="docutils"/>
<div class="section" id="references">
<h2>References</h2>
<div class="footnote-table">
<table class="footnote" id="id1"><tr>
<td class="label">[1]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id2">↩</a></td>
<td><a class="reference external" href="http://wordpress.com">http://wordpress.com</a></td>
</tr></table>
<table class="footnote" id="id3"><tr>
<td class="label">[2]</td>
<td class="fn-backrefs">↩<span class="fn-backref-list"> (<a class="fn-backref" href="#id4">1</a>, <a class="fn-backref" href="#id5">2</a>)</span></td>
<td><a class="reference external" href="http://getpelican.com">http://getpelican.com</a></td>
</tr></table>
<table class="footnote" id="id6"><tr>
<td class="label">[3]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id7">↩</a></td>
<td><a class="reference external" href="http://kernel.org">http://kernel.org</a></td>
</tr></table>
<table class="footnote" id="id8"><tr>
<td class="label">[4]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id9">↩</a></td>
<td><a class="reference external" href="https://github.com/dtzWill/wdtz">https://github.com/dtzWill/wdtz</a></td>
</tr></table>
<table class="footnote" id="id10"><tr>
<td class="label">[5]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id11">↩</a></td>
<td><a class="reference external" href="https://github.com/masterkoppa/Pelican-Themes/tree/master/bootstraped">https://github.com/masterkoppa/Pelican-Themes/tree/master/bootstraped</a></td>
</tr></table>
<table class="footnote" id="id12"><tr>
<td class="label">[6]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id13">↩</a></td>
<td><a class="reference external" href="http://bootswatch.com/cyborg/">http://bootswatch.com/cyborg/</a></td>
</tr></table>
<table class="footnote" id="id14"><tr>
<td class="label">[7]</td>
<td class="fn-backrefs"><a class="fn-backref" href="#id15">↩</a></td>
<td><a class="reference external" href="http://docutils.sourceforge.net/rst.html">http://docutils.sourceforge.net/rst.html</a></td>
</tr></table>
</div></div>