atom.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

  <title><![CDATA[Anjuke Engineering]]></title>
  <link href="http://arch.corp.anjuke.com/atom.xml" rel="self"/>
  <link href="http://arch.corp.anjuke.com/"/>
  <updated>2013-03-13T12:15:00+08:00</updated>
  <id>http://arch.corp.anjuke.com/</id>
  <author>
    <name><![CDATA[Anjuke Inc.]]></name>
    
  </author>
  <generator uri="http://octopress.org/">Octopress</generator>

  
  <entry>
    <title type="html"><![CDATA[使用安居客提供的PyPI镜像]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2013/03/01/using-pypi-provided-by-anjuke/"/>
    <updated>2013-03-01T10:52:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2013/03/01/using-pypi-provided-by-anjuke</id>
    <content type="html"><![CDATA[<p>主要提示：安居客现在拥有了自己的PyPI镜像，直接使用http://pypi.corp.anjuke.com/simple ，就可以节省重复去公网下载模块的流量与时间。</p>

<h2>Welcome to the Python World!</h2>

<p>这篇文章的目标读者是还刚接触Python以及安居客将要使用Python的各位开发者，强烈建议把文中提到的延伸信息都阅读一下，百利无害。</p>

<p>使用Python开发程序是一件轻松惬意的事情，它的第三方模块分发机制让开发者能够很方便快速的发布自己的代码、安装部署使用其他开发者做的模块。（详细请参考<a href="http://docs.python.org/2/install/index.html">安装Python模块</a>，<a href="http://docs.python.org/2/distutils/index.html">发布Python模块</a>）</p>

<p>关于Python打包的方式以及开源软加架构里打包的背景信息，可以阅读<a href="http://www.ituring.com.cn/article/19090">这篇内容</a>了解更多。</p>

<h2>如何安装第三方的Python模块</h2>

<p>标准的Python模块源码包里都会带上一个<code>setup.py</code>，它是Python的模块分发机制中的一部分，其中会定义模块的基本信息以及 <strong>依赖</strong> ，后者就是我们重点需要关注的问题。</p>

<p>通常我们只需要执行</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>python setup.py install</span></code></pre></td></tr></table></div></figure>


<p>即可完成安装。</p>

<h2>Python模块的分发</h2>

<p>传统模块的分发如果采用原始的人肉方式那就显得不够好了；所以Python社区有了PyPI（Python Package Index），官方的地址是<a href="https://pypi.python.org">pypi.python.org</a>，全世界各地还有几个<a href="http://www.pypi-mirrors.org">镜像</a>；以及配套使用的客户端程序，<code>setuptools</code>和<code>pip</code></p>

<p><strong>我们建议使用<code>pip</code></strong></p>

<p>前者一般在Python安装的时候随同一起分发，后者需要手工安装（<code>pythonbrew</code>里则是会自动一起安装）</p>

<p>对于想要使用的模块依赖，使用</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>easy_install foobar
</span><span class='line'>
</span><span class='line'>#或者
</span><span class='line'>pip install foobar</span></code></pre></td></tr></table></div></figure>


<p>它的过程就像是ubuntu下的<code>apt-get</code>，自动从配置的镜像（默认是官方）自动查找目标模块，下载，（编译）安装，整个过程轻松简单。</p>

<h2>为什么我们要提供安居客的镜像</h2>

<p>出于安居客未来架构发展的需要，我们在开发、生产环境的部署将会更多的采用自动化；项目的依赖是其中一个不可忽视的环节。对于Python来说，快速高效地安装依赖能提高效率，减少错误。</p>

<p>还有一点是，每次安装都从官方下载会造成流量和时间的浪费；更多的，因为某些大家都懂的原因造成一个项目依赖无法安装，很令人恼火。做镜像提高速度，减少问题也是让工程师快乐的一个体现。</p>

<h2>如何使用安居客的PyPI镜像</h2>

<p><strong>更详细的使用方法请参考<code>easy_install</code>和<code>pip</code>的文档</strong></p>

<h3>安装 (install)</h3>

<ul>
<li>easy_install</li>
</ul>


<p>直接使用easy_install的命令，带上参数指定</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>easy_install -i http://pypi.corp.anjuke.com/simple FOOBAR</span></code></pre></td></tr></table></div></figure>


<p>或者写配置文件<code>~/.pydistutils.cfg</code>，内容如下</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>[easy_install]
</span><span class='line'>index_url = http://pypi.corp.anjuke.com/simple</span></code></pre></td></tr></table></div></figure>


<ul>
<li>pip</li>
</ul>


<p>直接使用pip命令，带上参数</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>pip -i http://pypi.corp.anjuke.com/simple install FOOBAR</span></code></pre></td></tr></table></div></figure>


<p>或者写配置文件<code>~/.pip/pip.conf</code>，内容如下</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>[global]
</span><span class='line'>index-url = http://pypi.corp.anjuke.com/simple</span></code></pre></td></tr></table></div></figure>


<h3>搜索 (search)</h3>

<p><strong>暂未支持</strong></p>

<h2>最后</h2>

<p>我们希望为工程师、开发者提供更好的环境来支持开发工作。</p>

<p>如果大家在使用中有各种问题，意见和建议，欢迎联系我们。</p>

<p>Happy Hacking ;)</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Homebrew is just like gentoo portage]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2013/01/20/homebrew-is-just-like-gentoo-portage/"/>
    <updated>2013-01-20T14:28:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2013/01/20/homebrew-is-just-like-gentoo-portage</id>
    <content type="html"><![CDATA[<p><img class="right" src="http://0.gravatar.com/avatar/ce1e13bbf946c92e2abf740f8909bafa"></p>

<h2>一句话提示：</h2>

<blockquote><p>利用<code>brew versions</code>和<code>brew switch</code>命令也可以实现gentoo portage里eselect类似的版本切换功能!</p></blockquote>

<p>老规矩，喜欢听故事的继续看下去XDDD</p>

<!-- more -->


<h2>您使用何种包管理？</h2>

<p>MacOS下安装软件是件很惬意的事情，因为DMG或者PKG一路NEXT魔法即可(<strike>这么说Windows其实也是XDD</strike>)，卸载也不是什么难事。Linux下安装软件同样&#8230;也不能算是一件难事，因为我们有apt、pacman、emerge<strike>甚至是yum</strike>这样的包管理器，一条命令随叫随到。</p>

<p>那么有在Mac下像Linux那样安装软件包的工具吗？如今大家可能第一想到的是<a href="https://github.com/mxcl/homebrew">Homebrew</a>（以下简称brew），没错，在他之前就有大名鼎鼎的<a href="http://www.macports.org/">Macports</a>，号称和BSD的ports一样自动下载源代码编译安装的工具。那么为什么Homebrew如今却广受欢迎呢，我想了想</p>

<ul>
<li>支持最新的MacOS</li>
<li>包含着常用的大多数软件包</li>
<li>更新很快</li>
<li>够Cool! (使用Git方式发布，项目托管在Github上)</li>
<li><strike>是用ruby写的XDDD</strike></li>
</ul>


<p>brew里的软件是通过Formula方式写成一个打包文件，git同步后，让brew系统来读取这个Formula进行常见的下载、编译、安装过程。看起来和gentoo portage或者BSD的ports如出一辙(如果使用archlinux，可以理解成ABS)。</p>

<p>平时在使用系统的时候比较大的一项烦恼是，我有同一个软件，有不同的版本想要同时使用？如果软件包本身不妨碍多个版本共存那自然是皆大欢喜，不幸的是多数软件都不支持。进一步讲，我是一个开发者，需要引用不同的第三方库的时候这个问题怎么解决呢？</p>

<p>大家也许想到了开发语言自带的包管理器和版本控制工具，这方面的例子，rvm与gem/bundler首当其冲。假使您还不了解其中的重要性，推荐阅读一下<a href="http://www.12factor.net/dependencies">『The Twelve-Factor App』里关于项目依赖的部分</a>。其他的例子，还有python的setuptools/pip等等。自然的，<a href="http://blog.astrumfutura.com/2007/10/to-pear-or-not-to-pear-and-how-to-pear-anyway/">这个方面做的比较差的也有</a>。</p>

<p>更进一步讲，要是我使用的是系统的库呢，除了像<a href="https://github.com/CocoaPods/CocoaPods">CocoaPods</a>这种吸收了gem的新兴工具外，似乎走进了一个死胡同。其实不然，我们可以选择带有这样功能的包管理器或者系统！</p>

<h2>这就是为什么我喜爱gentoo与homebrew!</h2>

<p><a href="http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&amp;chap=1#doc_chap5">gentoo portage里有一个叫做slot的功能</a>，简单地解释一下，在portage的版本树里一个软件包可以有多个版本的ebuild文件，并把他们划分成不同地slot——例如python——大家都知道python目前有2.x和3.x的大版本区别，在portage里python就有几个slot（见下面的portage记录，版本之前括号里的就是slot），其中有2.7和3.2，他们属于同一个包下的多个slot版本；portage允许同一个包里多个slot并存，一般还会附赠一个<code>eselect-foobar</code>的包来提供版本的选择</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>[I] dev-lang/python
</span><span class='line'>     Available versions:
</span><span class='line'>  (2.2)   ~2.2-r8[1]
</span><span class='line'>  (2.5)   2.5.4-r4 ~2.5.4-r5
</span><span class='line'>  (2.6)   2.6.8 ~2.6.8-r1
</span><span class='line'>  (2.7)   2.7.3-r2 ~2.7.3-r3
</span><span class='line'>  (3.1)   3.1.5 ~3.1.5-r1
</span><span class='line'>  (3.2)   3.2.3 ~3.2.3-r1 ~3.2.3-r2
</span><span class='line'>  (3.3)   **3.3.0 **3.3.0-r1
</span><span class='line'>  
</span><span class='line'>     Installed versions:  2.7.3-r2(2.7)(03:02:44 PM 01/19/2013)(gdbm ipv6 ncurses readline sqlite ssl threads wide-unicode xml -berkdb -build -doc -elibc_uclibc -examples -tk -wininst) 3.2.3(3.2)(03:04:49 PM 01/19/2013)(gdbm ipv6 ncurses readline sqlite ssl threads wide-unicode xml -build -doc -elibc_uclibc -examples -tk -wininst)
</span><span class='line'>     Homepage:            http://www.python.org
</span><span class='line'>     Description:         A really great language
</span><span class='line'>
</span><span class='line'>[I] app-admin/eselect-python
</span><span class='line'>     Available versions:  20091230 20100321 ~20111108 **99999999
</span><span class='line'>     Installed versions:  20100321(02:18:04 PM 11/07/2012)
</span><span class='line'>     Homepage:            http://www.gentoo.org
</span><span class='line'>     Description:         Eselect module for management of multiple Python versions</span></code></pre></td></tr></table></div></figure>


<p>对于gentoo portage来说，不同的包版本安装的路径是不同的，例如2.7和3.2分别安装在<code>/usr/lib/python2.7</code>与<code>/usr/lib/python3.2</code>下，利用特殊的机制（eselect和软链等，python在gentoo下使用了python-wrapper）来实现多版本并存的功能。</p>

<p>不过可以留意到的是，在portage里同一个slot下也会有多个版本，他们之间是无法并存的XDD</p>

<h3>那么故事讲了这么多Homebrew到底有什么能耐呢？</h3>

<p>其实我前几天在自己的Mac姬上需要测试最新版本的zeromq 3.x，而同时另外有一个项目的zeromq是要求2.1.x版本的；按照常规做法，要么我自己手工编译两个lib分开放（意味着编译链接的FLAG都要自己设置，不开玩笑&#8230;），要么在开发项目A时重新安装3.x，开发项目B时重新安装2.1.x，岂不蛋疼？</p>

<p>让我们回想一下Homebrew的目录结构：官方建议安装在/usr/local下（因为MacOS不会使用这个目录）</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>├── CONTRIBUTING.md
</span><span class='line'>├── Cellar
</span><span class='line'>├── Library
</span><span class='line'>├── README.md
</span><span class='line'>├── bin
</span><span class='line'>├── etc
</span><span class='line'>├── include
</span><span class='line'>├── lib
</span><span class='line'>├── opt
</span><span class='line'>├── sbin
</span><span class='line'>├── share
</span><span class='line'>├── tmp
</span><span class='line'>└── var</span></code></pre></td></tr></table></div></figure>


<p>bin和lib下是常见的直接使用的文件，实际上是做了软链接到Cellar下具体某个包、某个版本下，例如</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>ls -lah lib/libzmq.a
</span><span class='line'>lrwxr-xr-x  1 aleiphoenix  wheel    36B Jan 16 15:54 lib/libzmq.a -&gt; ../Cellar/zeromq/2.1.11/lib/libzmq.a</span></code></pre></td></tr></table></div></figure>


<p>这下明白了吧，由于brew的实现已经是同一个包不同版本的分开放置，所以要支持多版本切换，只是写一个切换软链接的工具而已，而brew里已经自带了这个功能；并且这个功能比gentoo portage的slot更自由一些，因为任意版本之间都是可以并存的XDDD</p>

<ul>
<li>查看已经安装的版本 <code>brew info</code></li>
</ul>


<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>brew info zeromq
</span><span class='line'>zeromq: stable 3.2.2, HEAD
</span><span class='line'>http://www.zeromq.org/
</span><span class='line'>Depends on: pkg-config
</span><span class='line'>/usr/local/Cellar/zeromq/2.1.11 (41 files, 1.6M) *
</span><span class='line'>/usr/local/Cellar/zeromq/2.2.0 (41 files, 1.6M)
</span><span class='line'>/usr/local/Cellar/zeromq/3.2.2 (54 files, 2.2M)</span></code></pre></td></tr></table></div></figure>


<ul>
<li>查看所有版本 <code>brew versions</code></li>
</ul>


<p>这里有个技巧，由于brew的目录是用git管理的，所以对于Formula来说始终是一个文件（名），利用git命令把需要的版本checkout出来，就可以使用了XDDD</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>brew versions zeromq
</span><span class='line'>3.2.2    git checkout ab8de4b Library/Formula/zeromq.rb
</span><span class='line'>2.2.0    git checkout 6a2e6ef Library/Formula/zeromq.rb
</span><span class='line'>2.1.11   git checkout 497b13a Library/Formula/zeromq.rb
</span><span class='line'>2.1.10   git checkout 4c8ed3a Library/Formula/zeromq.rb
</span><span class='line'>2.1.9    git checkout 381c97f Library/Formula/zeromq.rb
</span><span class='line'>2.1.7    git checkout ed41f79 Library/Formula/zeromq.rb
</span><span class='line'>2.1.8    git checkout 8e045d5 Library/Formula/zeromq.rb
</span><span class='line'>2.1.6    git checkout 460a168 Library/Formula/zeromq.rb
</span><span class='line'>2.1.4    git checkout 83ed494 Library/Formula/zeromq.rb
</span><span class='line'>2.1.3    git checkout f4a925d Library/Formula/zeromq.rb
</span><span class='line'>2.1.2    git checkout 3017b39 Library/Formula/zeromq.rb
</span><span class='line'>2.1.1    git checkout 0476235 Library/Formula/zeromq.rb
</span><span class='line'>2.0.10   git checkout 00e1ae3 Library/Formula/zeromq.rb
</span><span class='line'>2.0.9    git checkout 0527b6f Library/Formula/zeromq.rb
</span><span class='line'>2.0.8    git checkout 1f252bf Library/Formula/zeromq.rb
</span><span class='line'>2.0.7    git checkout ce8d2f5 Library/Formula/zeromq.rb</span></code></pre></td></tr></table></div></figure>


<ul>
<li>切换至目标版本 <code>brew switch</code></li>
</ul>


<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>brew switch zeromq 3.2.2
</span><span class='line'>Cleaning /usr/local/Cellar/zeromq/2.1.11
</span><span class='line'>Cleaning /usr/local/Cellar/zeromq/2.2.0
</span><span class='line'>Cleaning /usr/local/Cellar/zeromq/3.2.2
</span><span class='line'>49 links created for /usr/local/Cellar/zeromq/3.2.2</span></code></pre></td></tr></table></div></figure>


<p>当然，这个并不表示brew一点问题也没有，比如某个libA引用的是特定版本的libB，在libB切换版本后libA可能就出错了。</p>

<blockquote><p>&#8220;It&#8217;s up to you, Mason. It&#8217;s all up to you.&#8221;</p></blockquote>

<p>所以XDDDD</p>

<p>总体而言，brew这个功能给开发者（<strike>不折腾会死星人</strike>）提供了莫大的便利。那天发现这个功能时，不禁感叹，『这不就是gentoo portage么！』 XDDDDDDDDDD</p>

<p>__END__</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Nginx热升级]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/12/23/nginx-live-upgrade/"/>
    <updated>2012-12-23T22:11:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/12/23/nginx-live-upgrade</id>
    <content type="html"><![CDATA[<p>系统管理员可以使用Nginx提供的信号机制来对其进行维护，比较常用的是<code>kill -HUP &lt;master pid&gt;</code>命令，它能通知Nginx使用新的配置文件启动工作进程，并逐个关闭旧进程，完成平滑切换。当需要对Nginx进行版本升级或增减模块时，为了不丢失请求，可以结合使用<code>USR2</code>、<code>WINCH</code>等信号进行平滑过度，达到热升级的目的。如果中途遇到问题，也能立刻回退至原版本。</p>

<h2>操作步骤</h2>

<p>1、备份原Nginx二进制文件；</p>

<p>2、编译新Nginx源码，安装路径需与旧版一致；</p>

<p>3、向主进程发送<code>USR2</code>信号，Nginx会启动一个新版本的master进程和工作进程，和旧版一起处理请求：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>prey:~ root# ps -ef|grep nginx
</span><span class='line'> 127     1   nginx: master process /usr/local/nginx-1.2.4/sbin/nginx
</span><span class='line'> 129   127   nginx: worker process
</span><span class='line'>prey:~ root# <span class="nb">kill</span> -USR2 127
</span><span class='line'>prey:~ root# ps -ef|grep nginx
</span><span class='line'> 127     1   nginx: master process /usr/local/nginx-1.2.4/sbin/nginx
</span><span class='line'> 129   127   nginx: worker process
</span><span class='line'>5180   127   nginx: master process /usr/local/nginx-1.2.4/sbin/nginx
</span><span class='line'>5182  5180   nginx: worker process
</span></code></pre></td></tr></table></div></figure>




<!-- more -->


<p>4、向原Nginx主进程发送<code>WINCH</code>信号，它会逐步关闭旗下的工作进程（主进程不退出），这时所有请求都会由新版Nginx处理：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>prey:~ root# <span class="nb">kill</span> -WINCH 127
</span><span class='line'>prey:~ root# ps -ef|grep nginx
</span><span class='line'> 127     1   nginx: master process /usr/local/nginx-1.2.4/sbin/nginx
</span><span class='line'>5180   127   nginx: master process /usr/local/nginx-1.2.4/sbin/nginx
</span><span class='line'>5182  5180   nginx: worker process
</span></code></pre></td></tr></table></div></figure>


<p>5、如果这时需要回退，可向原Nginx主进程发送<code>HUP</code>信号，它会重新启动工作进程， <strong>仍使用旧版配置文件</strong> 。尔后可以将新版Nginx进程杀死（使用<code>QUIT</code>、<code>TERM</code>、或者<code>KILL</code>）：</p>

<p>6、如果不需要回滚，可以将原Nginx主进程杀死，至此完成热升级。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>prey:~ root# <span class="nb">kill </span>127
</span><span class='line'>prey:~ root# ps -ef|grep nginx
</span><span class='line'>5180     1   nginx: master process /usr/local/nginx-1.2.4/sbin/nginx
</span><span class='line'>5182  5180   nginx: worker process
</span></code></pre></td></tr></table></div></figure>


<p>切换过程中，Nginx会将旧的<code>.pid</code>文件重命名为<code>.pid.oldbin</code>文件，并在旧进程退出后删除。</p>

<h2>原理简介</h2>

<h3>多进程模式下的请求分配方式</h3>

<p>Nginx默认工作在多进程模式下，即主进程（master process）启动后完成配置加载和端口绑定等动作，<code>fork</code>出指定数量的工作进程（worker process），这些子进程会持有监听端口的文件描述符（fd），并通过在该描述符上添加监听事件来接受连接（accept）。</p>

<h3>信号的接收和处理</h3>

<p>Nginx主进程在启动完成后会进入等待状态，负责响应各类系统消息，如SIGCHLD、SIGHUP、SIGUSR2等。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
</pre></td><td class='code'><pre><code class='c'><span class='line'><span class="c1">// src/os/unix/ngx_process_cycle.c</span>
</span><span class='line'><span class="kt">void</span>
</span><span class='line'><span class="nf">ngx_master_process_cycle</span><span class="p">(</span><span class="n">ngx_cycle_t</span> <span class="o">*</span><span class="n">cycle</span><span class="p">)</span>
</span><span class='line'><span class="p">{</span>
</span><span class='line'>    <span class="n">sigset_t</span>           <span class="n">set</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'>    <span class="n">sigemptyset</span><span class="p">(</span><span class="o">&amp;</span><span class="n">set</span><span class="p">);</span>
</span><span class='line'>    <span class="n">sigaddset</span><span class="p">(</span><span class="o">&amp;</span><span class="n">set</span><span class="p">,</span> <span class="n">SIGCHLD</span><span class="p">);</span>
</span><span class='line'>    <span class="n">sigaddset</span><span class="p">(</span><span class="o">&amp;</span><span class="n">set</span><span class="p">,</span> <span class="n">ngx_signal_value</span><span class="p">(</span><span class="n">NGX_RECONFIGURE_SIGNAL</span><span class="p">));</span>
</span><span class='line'>    <span class="n">sigaddset</span><span class="p">(</span><span class="o">&amp;</span><span class="n">set</span><span class="p">,</span> <span class="n">ngx_signal_value</span><span class="p">(</span><span class="n">NGX_CHANGEBIN_SIGNAL</span><span class="p">));</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">if</span> <span class="p">(</span><span class="n">sigprocmask</span><span class="p">(</span><span class="n">SIG_BLOCK</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">set</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">)</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>        <span class="n">ngx_log_error</span><span class="p">(</span><span class="n">NGX_LOG_ALERT</span><span class="p">,</span> <span class="n">cycle</span><span class="o">-&gt;</span><span class="n">log</span><span class="p">,</span> <span class="n">ngx_errno</span><span class="p">,</span>
</span><span class='line'>                      <span class="s">&quot;sigprocmask() failed&quot;</span><span class="p">);</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">for</span> <span class="p">(</span> <span class="p">;;</span> <span class="p">)</span> <span class="p">{</span>
</span><span class='line'>
</span><span class='line'>        <span class="n">sigsuspend</span><span class="p">(</span><span class="o">&amp;</span><span class="n">set</span><span class="p">);</span> <span class="c1">// 等待信号</span>
</span><span class='line'>
</span><span class='line'>        <span class="c1">// 信号回调函数定义在 src/os/unix/ngx_process.c 中，</span>
</span><span class='line'>        <span class="c1">// 它只负责设置全局变量，实际处理逻辑在这里。</span>
</span><span class='line'>        <span class="k">if</span> <span class="p">(</span><span class="n">ngx_change_binary</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>            <span class="n">ngx_change_binary</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</span><span class='line'>            <span class="n">ngx_log_error</span><span class="p">(</span><span class="n">NGX_LOG_NOTICE</span><span class="p">,</span> <span class="n">cycle</span><span class="o">-&gt;</span><span class="n">log</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="s">&quot;changing binary&quot;</span><span class="p">);</span>
</span><span class='line'>            <span class="n">ngx_new_binary</span> <span class="o">=</span> <span class="n">ngx_exec_new_binary</span><span class="p">(</span><span class="n">cycle</span><span class="p">,</span> <span class="n">ngx_argv</span><span class="p">);</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>上述代码中的<code>ngx_exec_new_binary</code>函数会调用<code>execve</code>系统函数运行一个新的Nginx主进程，并将当前监听端口的文件描述符通过环境变量的方式传递给新的主进程，这样新的主进程<code>fork</code>出的子进程就同样能够对监听端口添加事件回调，接受连接，从而使得新老Nginx共同处理用户请求。</p>

<h2>其它方式</h2>

<p>除了使用上述方式进行Nginx热升级，还可以选择以下两种方式：</p>

<h3>Keepalived主备切换</h3>

<p>Nginx用作LB时一般会用<a href="http://www.keepalived.org/">Keepalived</a>做热备，即DNS解析指向一个虚拟IP（VIP），主备服务器上分别启动Keepalived进程，当Master健康检查失败，Slave会自动抢夺VIP，完成切换。</p>

<p>在进行热升级时就可以使用这种方式，在Slave上进行Nginx升级，然后关闭Master的Keepalived进程，完成VIP的漂移。测试完成后可以对继续对Master进行升级操作，或选择回滚。</p>

<h3>Tengine动态模块</h3>

<p>当需要增加Nginx模块时，必须对Nginx源码进行重新编译，然后采用上面提到的方式进行热升级。Nginx官网上说未来并无打算增加动态模块加载的功能，至少1.x中不会。</p>

<p>如果你愿意使用<a href="http://tengine.taobao.org/">Tengine</a>，淘宝开发的一个Nginx分支，它提供了动态模块加载（DSO）功能。这里简单介绍一下使用方法：</p>

<h4>动态添加内部模块</h4>

<p>以<code>http_sub_module</code>为例，到Nginx源码目录执行以下命令：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>./configure --with-http_sub_module<span class="o">=</span>shared
</span><span class='line'><span class="nv">$ </span>make
</span><span class='line'><span class="nv">$ </span>sudo make dso_install
</span></code></pre></td></tr></table></div></figure>


<p>它会将编译好的<code>ngx_http_sub_filter_module.so</code>文件复制到<code>/usr/local/nginx/modules</code>目录下。随后在Nginx配置文件中的最外层添加以下内容：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>dso <span class="o">{</span>
</span><span class='line'>    load ngx_http_sub_filter_module.so;
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>执行<code>sbin/nginx -s reload</code>，就能完成模块的加载。</p>

<h4>第三方模块</h4>

<p>对于第三方模块，Tengine提供了<code>dso_tool</code>命令，能够一步安装：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>sbin/dso_tool --add-module<span class="o">=</span>/home/dso/lua-nginx-module
</span></code></pre></td></tr></table></div></figure>


<h2>Nginx信号汇总</h2>

<p>以下内容译自：http://wiki.nginx.org/CommandLine</p>

<h3>主进程支持的信号</h3>

<ul>
<li><code>TERM</code>, <code>INT</code>: 立刻退出</li>
<li><code>QUIT</code>: 等待工作进程结束后再退出</li>
<li><code>KILL</code>: 强制终止进程</li>
<li><code>HUP</code>: 重新加载配置文件，使用新的配置启动工作进程，并逐步关闭旧进程。</li>
<li><code>USR1</code>: 重新打开日志文件</li>
<li><code>USR2</code>: 启动新的主进程，实现热升级</li>
<li><code>WINCH</code>: 逐步关闭工作进程</li>
</ul>


<h3>工作进程支持的信号</h3>

<ul>
<li><code>TERM</code>, <code>INT</code>: 立刻退出</li>
<li><code>QUIT</code>: 等待请求处理结束后再退出</li>
<li><code>USR1</code>: 重新打开日志文件</li>
</ul>


<h3>nginx -s signal 支持的信号</h3>

<ul>
<li><code>stop</code>: 等价于<code>TERM</code>, <code>INT</code></li>
<li><code>quit</code>: <code>QUIT</code></li>
<li><code>reopen</code>: <code>USR1</code></li>
<li><code>reload</code>: <code>HUP</code></li>
</ul>


<h3>使用方法</h3>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>sbin/nginx -s reload
</span><span class='line'><span class="nv">$ </span><span class="nb">kill</span> -HUP <span class="k">$(</span>cat logs/nginx.pid<span class="k">)</span>
</span></code></pre></td></tr></table></div></figure>



]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Clojure实战(3)：使用Noir框架开发博客(下)]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/12/16/cia-noir-3/"/>
    <updated>2012-12-16T20:20:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/12/16/cia-noir-3</id>
    <content type="html"><![CDATA[<h2>Session和Cookie</h2>

<p>做网络编程的人肯定对这两个概念不陌生，因此这里就不介绍它们的定义和作用了。我们要实现的需求也很简单：用户通过一个表单登录，在当前窗口中保持登录状态，并可以选择“记住我”来免去关闭并新开窗口之后的重登录。显然，前者使用Session，后者使用Cookie。下面我们就来看Noir对这两者的支持。</p>

<h3>Session</h3>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="nf">require</span> <span class="ss">&#39;noir.session</span><span class="p">)</span>
</span><span class='line'><span class="p">(</span><span class="nf">noir.session/put!</span> <span class="ss">:username</span> <span class="s">&quot;john&quot;</span><span class="p">)</span>
</span><span class='line'><span class="p">(</span><span class="nf">noir.session/get</span> <span class="ss">:username</span> <span class="s">&quot;nobody&quot;</span><span class="p">)</span>
</span><span class='line'><span class="p">(</span><span class="nf">noir.session/clear!</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>很简单的API。注意<code>put!</code>函数中的<code>!</code>，和之前遇到的<code>?</code>一样，这种特殊字符是合法的函数名，但<code>!</code>习惯用来表示该方法会改变某个对象的状态，这里<code>put!</code>就表示会改变Session的状态。</p>

<p>Noir还提供了一种“闪信（Flash）”机制，主要用于在页面跳转之间暂存消息。如用户登录后会跳转到首页，如果想在首页显示“登录成功”的信息，就需要用到闪信了。闪信的API也放置在<code>noir.session</code>命名空间下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="nf">noir.session/flash-put!</span> <span class="s">&quot;登录成功&quot;</span><span class="p">)</span>
</span><span class='line'><span class="p">(</span><span class="nf">noir.session/flash-get</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>闪信的生命周期是一次请求，即在设置了闪信后的下一个请求中，可以多次<code>flash-get</code>，但再下一次请求就获取不到值了。</p>

<!-- more -->


<h3>Cookie</h3>

<p>Cookie的API示例如下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="nf">require</span> <span class="ss">&#39;noir.cookies</span><span class="p">)</span>
</span><span class='line'><span class="p">(</span><span class="nf">noir.cookies/put!</span> <span class="ss">:user_id</span> <span class="p">(</span><span class="nb">str </span><span class="mi">1</span><span class="p">))</span>
</span><span class='line'><span class="p">(</span><span class="nf">noir.cookies/get</span> <span class="ss">:user_id</span><span class="p">)</span>
</span><span class='line'><span class="p">(</span><span class="nf">noir.cookies/put!</span> <span class="ss">:tracker</span> <span class="p">{</span><span class="ss">:value</span> <span class="p">(</span><span class="nb">str </span><span class="mi">29649</span><span class="p">)</span> <span class="ss">:path</span> <span class="s">&quot;/&quot;</span> <span class="ss">:max-age</span> <span class="mi">3600</span><span class="p">})</span>
</span></code></pre></td></tr></table></div></figure>


<p>需要注意的是，<code>put!</code>函数只支持字符串类型；对于Cookie超时时间的设置，一种是上面所写的多少秒过期，另一种是传入一个DateTime对象。对于时间日期的处理，Java自带的类库可能不太好用，这里推荐<a href="http://joda-time.sourceforge.net/">Joda Time</a>，它有更丰富的功能和更友善的API。</p>

<h2>登录页面</h2>

<p>这里我们跳过注册页面，因为它实现的功能和新建一篇文章很相近，所以读者可以自己完成。我们假定用户信息表的格式如下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='sql'><span class='line'><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="o">`</span><span class="k">user</span><span class="o">`</span> <span class="p">(</span>
</span><span class='line'>  <span class="o">`</span><span class="n">id</span><span class="o">`</span> <span class="nb">int</span><span class="p">(</span><span class="mi">11</span><span class="p">)</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="n">AUTO_INCREMENT</span><span class="p">,</span>
</span><span class='line'>  <span class="o">`</span><span class="n">username</span><span class="o">`</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">255</span><span class="p">)</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
</span><span class='line'>  <span class="o">`</span><span class="n">password</span><span class="o">`</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">32</span><span class="p">)</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
</span><span class='line'>  <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="p">(</span><span class="o">`</span><span class="n">id</span><span class="o">`</span><span class="p">)</span>
</span><span class='line'><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>其中password字段保存的是密码的MD5值（32位16进制字符串）。Clojure中没有提供专门的类库，因此需要调用Java来实现。下文会贴出它的实现代码。</p>

<p>我们重点来看对登录页面表单的处理。新建<code>src/blog/views/login.clj</code>文件，添加对<code>/login</code>的路由，显示一个包含用户名、密码、以及“记住我”复选框的表单。用户提交后，若验证成功，会跳转至<code>/whoami</code>页面，用来显示保存在session或者cookie中的信息。以下是关键代码：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="nf">defpage</span> <span class="p">[</span><span class="ss">:post</span> <span class="s">&quot;/login&quot;</span><span class="p">]</span> <span class="p">{</span><span class="ss">:as</span> <span class="nv">forms</span><span class="p">}</span>
</span><span class='line'>  <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nv">userid</span> <span class="p">(</span><span class="nf">model-user/get-id</span> <span class="p">(</span><span class="ss">:username</span> <span class="nv">forms</span><span class="p">)</span> <span class="p">(</span><span class="ss">:password</span> <span class="nv">forms</span><span class="p">))]</span>
</span><span class='line'>    <span class="p">(</span><span class="k">if </span><span class="nv">userid</span>
</span><span class='line'>      <span class="p">(</span><span class="k">do </span><span class="p">(</span><span class="nf">session/put!</span> <span class="ss">:userid</span> <span class="nv">userid</span><span class="p">)</span>
</span><span class='line'>          <span class="p">(</span><span class="nf">session/put!</span> <span class="ss">:username</span> <span class="p">(</span><span class="ss">:username</span> <span class="nv">forms</span><span class="p">))</span>
</span><span class='line'>          <span class="p">(</span><span class="nb">when </span><span class="p">(</span><span class="nb">= </span><span class="p">(</span><span class="ss">:remember-me</span> <span class="nv">forms</span><span class="p">)</span> <span class="s">&quot;1&quot;</span><span class="p">)</span> <span class="c1">; “记住我”复选框</span>
</span><span class='line'>            <span class="p">(</span><span class="nf">cookies/put!</span> <span class="ss">:userid</span> <span class="p">{</span><span class="ss">:value</span> <span class="p">(</span><span class="nb">str </span><span class="nv">userid</span><span class="p">)</span> <span class="ss">:max-age</span> <span class="mi">86400</span><span class="p">})</span> <span class="c1">; 保存登录状态，时限1天。</span>
</span><span class='line'>            <span class="p">(</span><span class="nf">cookies/put!</span> <span class="ss">:username</span> <span class="p">{</span><span class="ss">:value</span> <span class="p">(</span><span class="ss">:username</span> <span class="nv">forms</span><span class="p">)</span> <span class="ss">:max-age</span> <span class="mi">86400</span><span class="p">}))</span>
</span><span class='line'>          <span class="p">(</span><span class="nf">response/redirect</span> <span class="s">&quot;/whoami&quot;</span><span class="p">))</span> <span class="c1">; noir.response/redirect 302跳转</span>
</span><span class='line'>      <span class="p">(</span><span class="nf">render</span> <span class="s">&quot;/login&quot;</span> <span class="nv">forms</span><span class="p">))))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="nf">defpage</span> <span class="s">&quot;/whoami&quot;</span> <span class="p">[]</span> <span class="c1">; 先检测Session，再检测Cookie。</span>
</span><span class='line'>  <span class="p">(</span><span class="nb">if-let </span><span class="p">[</span><span class="nv">userid</span> <span class="p">(</span><span class="nf">session/get</span> <span class="ss">:userid</span><span class="p">)]</span>
</span><span class='line'>    <span class="p">(</span><span class="nf">session/get</span> <span class="ss">:username</span><span class="p">)</span>
</span><span class='line'>    <span class="p">(</span><span class="nb">if-let </span><span class="p">[</span><span class="nv">userid</span> <span class="p">(</span><span class="nf">cookies/get</span> <span class="ss">:userid</span><span class="p">)]</span>
</span><span class='line'>      <span class="p">(</span><span class="nf">do</span>
</span><span class='line'>        <span class="p">(</span><span class="nf">session/put!</span> <span class="ss">:userid</span> <span class="nv">userid</span><span class="p">)</span>
</span><span class='line'>        <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nv">username</span> <span class="p">(</span><span class="nf">cookies/get</span> <span class="ss">:username</span><span class="p">)]</span>
</span><span class='line'>          <span class="p">(</span><span class="nf">session/put!</span> <span class="ss">:username</span> <span class="nv">username</span><span class="p">)</span>
</span><span class='line'>          <span class="nv">username</span><span class="p">))</span>
</span><span class='line'>      <span class="s">&quot;unknown&quot;</span><span class="p">)))</span>
</span></code></pre></td></tr></table></div></figure>


<p>其中<code>if-let</code>和以下代码是等价的，类似的有<code>when-let</code>。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nv">userid</span> <span class="p">(</span><span class="nf">session/get</span> <span class="ss">:userid</span><span class="p">)]</span>
</span><span class='line'>  <span class="p">(</span><span class="k">if </span><span class="nv">userid</span>
</span><span class='line'>    <span class="p">(</span><span class="k">do </span><span class="nv">...</span><span class="p">)</span>
</span><span class='line'>    <span class="s">&quot;unkown&quot;</span><span class="p">))</span>
</span></code></pre></td></tr></table></div></figure>


<p>对用户表的操作我们放到<code>src/blog/models/user.clj</code>文件中：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">blog.models.user</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">clojure.java.jdbc</span> <span class="ss">:as</span> <span class="nv">sql</span><span class="p">]</span>
</span><span class='line'>            <span class="p">[</span><span class="nv">blog.util</span> <span class="ss">:as</span> <span class="nv">util</span><span class="p">])</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:use</span> <span class="p">[</span><span class="nv">blog.database</span> <span class="ss">:only</span> <span class="p">[</span><span class="nv">db-spec</span><span class="p">]]))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="kd">defn </span><span class="nv">get-id</span> <span class="p">[</span><span class="nv">username</span> <span class="nv">password</span><span class="p">]</span>
</span><span class='line'>  <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nv">password-md5</span> <span class="p">(</span><span class="nf">util/md5</span> <span class="nv">password</span><span class="p">)]</span>
</span><span class='line'>    <span class="p">(</span><span class="nf">sql/with-connection</span> <span class="nv">db-spec</span>
</span><span class='line'>      <span class="p">(</span><span class="nf">sql/with-query-results</span> <span class="nv">rows</span>
</span><span class='line'>        <span class="p">[</span><span class="s">&quot;SELECT `id` FROM `user` WHERE `username` = ? AND `password` = ?&quot;</span>
</span><span class='line'>         <span class="nv">username</span> <span class="nv">password-md5</span><span class="p">]</span> <span class="c1">; 不要采用直接拼接字符串的方式，有SQL注入的危险。</span>
</span><span class='line'>        <span class="p">(</span><span class="ss">:id</span> <span class="p">(</span><span class="nb">first </span><span class="nv">rows</span><span class="p">))))))</span>
</span></code></pre></td></tr></table></div></figure>


<p>最后，我们将MD5加密这类的函数放到<code>src/blog/util.clj</code>文件中：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">blog.util</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:import</span> <span class="nv">java.security.MessageDigest</span>
</span><span class='line'>           <span class="nv">java.math.BigInteger</span><span class="p">))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="kd">defn </span><span class="nv">md5</span> <span class="p">[</span><span class="nv">s</span><span class="p">]</span>
</span><span class='line'>  <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nv">algorithm</span> <span class="p">(</span><span class="nf">MessageDigest/getInstance</span> <span class="s">&quot;MD5&quot;</span><span class="p">)</span>
</span><span class='line'>        <span class="nv">size</span> <span class="p">(</span><span class="nb">* </span><span class="mi">2</span> <span class="p">(</span><span class="nf">.getDigestLength</span> <span class="nv">algorithm</span><span class="p">))</span>
</span><span class='line'>        <span class="nv">raw</span> <span class="p">(</span><span class="nf">.digest</span> <span class="nv">algorithm</span> <span class="p">(</span><span class="nf">.getBytes</span> <span class="nv">s</span><span class="p">))</span>
</span><span class='line'>        <span class="nv">sig</span> <span class="p">(</span><span class="nf">.toString</span> <span class="p">(</span><span class="nf">BigInteger.</span> <span class="mi">1</span> <span class="nv">raw</span><span class="p">)</span> <span class="mi">16</span><span class="p">)</span>
</span><span class='line'>        <span class="nv">padding</span> <span class="p">(</span><span class="nb">apply str </span><span class="p">(</span><span class="nb">repeat </span><span class="p">(</span><span class="nb">- </span><span class="nv">size</span> <span class="p">(</span><span class="nb">count </span><span class="nv">sig</span><span class="p">))</span> <span class="s">&quot;0&quot;</span><span class="p">))]</span>
</span><span class='line'>    <span class="p">(</span><span class="nb">str </span><span class="nv">padding</span> <span class="nv">sig</span><span class="p">)))</span>
</span></code></pre></td></tr></table></div></figure>


<p><code>padding</code>的作用是当计算得到的MD5字符串不足32位时做补零的操作。如何得到一个包含N个&#8221;0&#8221;的字符串？这就是<code>(apply...)</code>那串代码做的工作。简单来说，<code>repeat</code>函数会返回一个序列，<code>apply</code>函数首先使用第1、第2个元素作为参数调用<code>str</code>函数，然后将执行结果和第3个元素作为参数调用<code>str</code>，依此类推。因此，<code>(apply str [1 2 3])</code>等价于<code>(str (str 1 2) 3)</code>。<code>clojure.string/join</code>提供了将序列连接为字符串的功能，用法是<code>(clojure.string/join (repeat ...))</code>，查看它的源码<code>(source clojure.string/join)</code>可以发现，它实质上也是采用了<code>apply</code>函数。</p>

<p>序列是Clojure的一个很重要的数据结构，有多种函数和惯用法，需要逐步积累这些知识。</p>

<h2>中间件</h2>

<p>如果需要在程序的多个地方获取用户的登录状态，可以将上述<code>/whoami</code>中的方法封装成函数，但是每次都要执行一次似乎有些冗余，因此我们可以将它放到中间件（Middleware）中。</p>

<p>中间件是<a href="http://en.wikipedia.org/wiki/Web_Server_Gateway_Interface">WSGI</a>类的网站程序中很重要的特性。如果将用户的一次访问分解成<code>请求-&gt;处理1-&gt;处理2-&gt;应答</code>，那么中间件就是其中的“处理”部分，可以增加任意多个。Noir的很多功能，像路由、Session等，都是通过中间件的形式进行组织的。</p>

<p>以下是一个空的中间件代码：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">...</span>
</span><span class='line'>    <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">noir.server</span> <span class="ss">:as</span> <span class="nv">server</span><span class="p">]))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="kd">defn </span><span class="nv">my-middleware</span> <span class="p">[</span><span class="nv">handler</span><span class="p">]</span>
</span><span class='line'>  <span class="p">(</span><span class="k">fn </span><span class="p">[</span><span class="nv">request</span><span class="p">]</span>
</span><span class='line'>    <span class="p">(</span><span class="nf">handler</span> <span class="nv">request</span><span class="p">)))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="nf">erver/add-middleware</span> <span class="nv">my-middleware</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>上述代码添加到<code>src/blog/server.clj</code>中可以直接运行，只是这个中间件没有做任何工作。中间件是一个函数，返回值是一个匿名函数（<code>defn</code>是基于<code>fn</code>的，详情可见<code>(doc defn)</code>）。<code>handler</code>参数则是前一个中间件返回的匿名函数，<code>request</code>是用户发送过来的请求（map形式）。这些中间件组合起来就成为了一条处理链。<code>add-middleware</code>则是Noir定义的函数，将用户自定义的中间件添加到处理链中。</p>

<p>下面我们就写这样一个中间件，每次请求时都去检测Session和Cookie中是否包含用户的登录信息，并将该信息放到<code>request</code>的map中：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">defn </span><span class="nv">authenticate</span> <span class="p">[</span><span class="nv">handler</span><span class="p">]</span>
</span><span class='line'>  <span class="p">(</span><span class="k">fn </span><span class="p">[</span><span class="nv">request</span><span class="p">]</span>
</span><span class='line'>    <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nv">user</span> <span class="p">(</span><span class="nb">if-let </span><span class="p">[</span><span class="nv">userid</span> <span class="p">(</span><span class="nf">session/get</span> <span class="ss">:userid</span><span class="p">)]</span>
</span><span class='line'>                <span class="p">[</span><span class="nv">userid</span> <span class="p">(</span><span class="nf">session/get</span> <span class="ss">:username</span><span class="p">)]</span>
</span><span class='line'>                <span class="p">(</span><span class="nb">when-let </span><span class="p">[</span><span class="nv">userid</span> <span class="p">(</span><span class="nf">cookies/get</span> <span class="ss">:userid</span><span class="p">)]</span>
</span><span class='line'>                  <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nv">username</span> <span class="p">(</span><span class="nf">cookies/get</span> <span class="ss">:username</span><span class="p">)]</span>
</span><span class='line'>                    <span class="p">(</span><span class="nf">do</span>
</span><span class='line'>                      <span class="p">(</span><span class="nf">session/put!</span> <span class="ss">:userid</span> <span class="nv">userid</span><span class="p">)</span>
</span><span class='line'>                      <span class="p">(</span><span class="nf">session/put!</span> <span class="ss">:username</span> <span class="nv">username</span><span class="p">)</span>
</span><span class='line'>                      <span class="p">[</span><span class="nv">userid</span> <span class="nv">username</span><span class="p">]))))</span>
</span><span class='line'>          <span class="nv">req</span> <span class="p">(</span><span class="k">if </span><span class="nv">user</span>
</span><span class='line'>                <span class="p">(</span><span class="nb">assoc </span><span class="nv">request</span> <span class="ss">:user</span> <span class="p">(</span><span class="nb">zipmap </span><span class="p">[</span><span class="ss">:userid</span> <span class="ss">:username</span><span class="p">]</span> <span class="nv">user</span><span class="p">))</span>
</span><span class='line'>                <span class="nv">request</span><span class="p">)]</span>
</span><span class='line'>      <span class="p">(</span><span class="nf">handler</span> <span class="nv">req</span><span class="p">))))</span>
</span></code></pre></td></tr></table></div></figure>


<p>这段代码中对于session和cookies的调用和上面没有差异，比较陌生的可能是<code>assoc</code>和<code>zipmap</code>方法，他们都是用来操作map数据类型的：前者会向一个map对象添加键值，并返回一个新的map；后者则会接收两个序列作为参数，两两组合成一个map并返回。</p>

<p>这样我们就能将<code>/whoami</code>的代码修改为：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">...</span>
</span><span class='line'>    <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">noir.request</span> <span class="ss">:as</span> <span class="nv">request</span><span class="p">]))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="nf">defpage</span> <span class="s">&quot;/whoami&quot;</span> <span class="p">[]</span>
</span><span class='line'>         <span class="p">(</span><span class="nb">if-let </span><span class="p">[</span><span class="nv">user</span> <span class="p">(</span><span class="ss">:user</span> <span class="p">(</span><span class="nf">request/ring-request</span><span class="p">))]</span>
</span><span class='line'>           <span class="p">(</span><span class="ss">:username</span> <span class="nv">user</span><span class="p">)</span>
</span><span class='line'>           <span class="s">&quot;unkown&quot;</span><span class="p">))</span>
</span></code></pre></td></tr></table></div></figure>


<p>其中，<code>ring-request</code>用来获得用户的<code>request</code>map对象。</p>

<h2>程序发布</h2>

<p>这里介绍三种Web应用程序的发布方式。</p>

<h3>直接使用Leiningen</h3>

<p>如果服务器上安装有<code>lein</code>环境，则可以直接调用它来启动程序。只有一点需要注意，因为在默认情况下，<code>lein run</code>启动的程序会被包装在Leiningen的JVM中，这样会占用一些额外的内存，同时引起一些<code>stdin</code>方面的问题。解决方法是使用<code>lein trampoline run</code>命令来启动程序，这样Leiningen为程序启动一个独立的JVM，并退出自己的JVM。</p>

<h3>编译为独立Jar包</h3>

<p><code>lein uberjar</code>命令可以将项目编译后的代码及其所有的依赖包打入一个Jar文件中，和Maven的assembly插件类似。需要注意的是，Clojure文件在默认情况下是不会生成类文件的，而是在运行时进行解析。这样一来，当使用<code>java -jar</code>命令执行时会提示找不到类定义的错误。解决方法是为包含入口函数的模块生成类文件，需要在<code>src/blog/server.clj</code>的<code>ns</code>声明中添加<code>gen-class</code>标识：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">blog.server</span>
</span><span class='line'>    <span class="nv">...</span>
</span><span class='line'>    <span class="p">(</span><span class="ss">:gen-class</span><span class="p">))</span>
</span></code></pre></td></tr></table></div></figure>


<p>然后就能打包运行了：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>lein uberjar
</span><span class='line'><span class="nv">$ </span>java -jar blog-0.1.0-SNAPSHOT-standalone.jar
</span><span class='line'>2012-12-23 00:07:47.417:INFO::jetty-6.1.x
</span><span class='line'>2012-12-23 00:07:47.430:INFO::Started SocketConnector@0.0.0.0:8080
</span></code></pre></td></tr></table></div></figure>


<p>可以在程序前部署一个Nginx代理做转发，配置方法就不在这里赘述了。</p>

<h3>使用Tomcat</h3>

<p>以上两种方法使用的都是Jetty这个Web容器，虽然比较方便，但在生产环境中我们更倾向于使用Tomcat。</p>

<p>对于Tomcat的安装这里不做讲解，读者可以到<a href="http://tomcat.apache.org/">Tomcat官网</a>查阅。</p>

<p>Clojure代码也需要做一些修改，我们需要提供一个接口供Tomcat调用，也就是<code>Handler</code>。在<code>src/blog/server.clj</code>中添加以下代码：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="k">def </span><span class="nv">handler</span> <span class="p">(</span><span class="nf">server/gen-handler</span>
</span><span class='line'>               <span class="p">{</span><span class="ss">:mode</span> <span class="ss">:prod</span>,
</span><span class='line'>                <span class="ss">:ns</span> <span class="ss">&#39;blog</span><span class="p">}))</span>
</span></code></pre></td></tr></table></div></figure>


<p><code>gen-handler</code>是Noir的函数，用来生成一个<code>Handler</code>。<code>'blog</code>前的单引号大家应该还有印象，它表示命名空间。</p>

<p><code>server.clj</code>还有一项内容需要修改：删除<code>load-views</code>，改为显式的<code>require</code>，这样才能保证在编译期间就加载路由配置，Tomcat才会认可。代码如下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">...</span>
</span><span class='line'>    <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">blog.views</span> <span class="nv">welcome</span> <span class="nv">article</span><span class="p">]))</span>
</span><span class='line'>
</span><span class='line'><span class="c1">; (server/load-views &quot;src/blog/views&quot;)</span>
</span></code></pre></td></tr></table></div></figure>


<p>和<code>uberjar</code>类似，我们需要使用<code>uberwar</code>来打包成一个包含所有依赖项的war包。不过这个工具是由一个Leiningen插件提供的：<code>lein-ring</code>，安装过程和<code>lein-noir</code>类似，首先在<code>project.clj</code>添加dev依赖，然后执行<code>lein deps</code>安装。要使上述<code>handler</code>生效，<code>project.clj</code>中还需要增加一项名为<code>:ring</code>的配置：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">defproject </span><span class="nv">blog</span> <span class="nv">...</span>
</span><span class='line'>            <span class="nv">...</span>
</span><span class='line'>            <span class="ss">:dev-dependencies</span> <span class="p">[</span><span class="nv">...</span>
</span><span class='line'>                               <span class="p">[</span><span class="nv">lein-ring</span> <span class="s">&quot;0.7.5&quot;</span><span class="p">]]</span>
</span><span class='line'>            <span class="ss">:ring</span> <span class="p">{</span><span class="ss">:handler</span> <span class="nv">blog.server/handler</span><span class="p">})</span>
</span></code></pre></td></tr></table></div></figure>


<p>执行<code>lein ring uberwar</code>命令，将生成的war包放置到Tomcat的webapps目录中，命名为ROOT.war，也可以设置<a href="http://tomcat.apache.org/tomcat-7.0-doc/virtual-hosting-howto.html">Virtual Hosting</a>。片刻后，Tomcat会应用这个新的程序，我们就能在浏览器中访问了。</p>

<h2>发布至云端Heroku</h2>

<p>最后，我们来尝试将这个博客程序部署到线上环境中。如今云计算已经非常流行，有许多优秀的<a href="http://en.wikipedia.org/wiki/Platform_as_a_service">PaaS</a>平台，<a href="http://www.heroku.com">Heroku</a>就是其中之一。在Heroku上部署一个小型的应用是完全免费的，这里我们简述一下步骤，更详细的操作方法可以参考它的<a href="https://devcenter.heroku.com/articles/clojure">帮助文档</a>。</p>

<ul>
<li>登录Heroku网站并注册账号；</li>
<li>安装<a href="https://toolbelt.heroku.com/">Tookbelt</a>，从而能在命令行中使用<code>heroku</code>命令；</li>
<li>执行<code>heroku login</code>命令，输入账号密码，完成验证；</li>
<li>新建<code>src/Procfile</code>文件，输入<code>web: lein trampoline run blog.server</code>；</li>
<li>执行<code>foreman start</code>命令，可以在本地测试程序；</li>
<li>执行<code>heroku create</code>，Heroku会为你分配一个空间；</li>
<li>执行<code>git push heroku master</code>，将本地代码推送至云端，可以看到编译信息，并得到一个URL，通过它就能访问我们的应用程序了。</li>
</ul>


<p>以上步骤省略了数据库的配置，读者可以自行到<a href="https://addons.heroku.com/cleardb">Heroku ClearDB</a>页面查看配置方法。</p>

<h2>小结</h2>

<p>至此我们完成了对Noir网站开发框架的简介，也完成了对Clojure这门语言的入门介绍。不过《Clojure实战》系列还远没有结束，下一章开始我们会进入Clojure语言更擅长的领域——计算。我们会陆续介绍如何使用Clojure编写<a href="http://hadoop.apache.org">Hadoop</a> MapReduce脚本、编写<a href="http://www.storm-project.net">Storm</a> Topology、以及如何使用<a href="http://incanter.org/">Incanter</a>进行可视化数据分析。不过在此之前，我强烈建议读者能够回头看看第一章中提到的几个Clojure教程，这样能对Clojure语言的整体架构有一个印象，接下来的学习才会更为顺畅。</p>

<h3>PS</h3>

<p>在撰写这份Noir框架教程时，Noir作者宣布停止对Noir的开发和维护，鼓励开发者转而使用Ring+Compojure+lib-noir的方式进行开发。这对我们并无太大影响，毕竟我们只是利用Noir来学习Clojure，而且前文提过Noir本身就是基于Ring和Compojure这两个类库的，迁移起来非常方便，我会为此再写一篇博客的。</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Clojure实战(2)：使用Noir框架开发博客(中)]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/12/08/cia-noir-2/"/>
    <updated>2012-12-08T12:09:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/12/08/cia-noir-2</id>
    <content type="html"><![CDATA[<h2>在Eclipse中编写Clojure代码</h2>

<p>从这章起我们就要开始真正的编码了。Vim可能是很多程序员的选择，但如果你像我一样更喜欢GUI界面，那就来看看如何在Eclipse中编写Clojure代码吧。</p>

<h3>安装Eclipse插件</h3>

<p>Eclipse提供了一个Clojure插件：CounterClockwise，可以用来编写Clojure代码，进行语法高亮、调试等操作。打开Eclipse的Market Place，搜索counterclockwise关键字，点击Install即可。</p>

<h3>将Leiningen项目导入Eclipse</h3>

<p>由于CounterClockwise插件并没有默认使用Leiningen来管理项目，因此需要做一些额外的工作。</p>

<p>在使用<code>lein new</code>命令创建项目后，在project.clj文件中增加如下一行：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">defproject </span><span class="nv">...</span>
</span><span class='line'>            <span class="ss">:dev-dependencies</span> <span class="p">[[</span><span class="nv">lein-eclipse</span> <span class="s">&quot;1.0.0&quot;</span><span class="p">]]</span>
</span><span class='line'>            <span class="nv">...</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>然后依次执行<code>lein deps</code>和<code>lein eclipse</code>，会看到项目根目录下生成了.project和.classpath文件。然后就可以进入Eclipse导入这个项目了。如果使用Git进行版本控制，<code>lein</code>已经为你生成好了.gitignore文件。执行了<code>git init</code>后，就能在Eclilpse中选择Share Project菜单项，进行可视化的版本控制。</p>

<h2>使用表单</h2>

<p>我们现在需要编写一个新建文章的功能，它是一个简单的页面，页面上有“标题”和“内容”两个文本框，并有一个“提交”按钮。</p>

<p>在src/blog/views目录下新建一个文件article.clj，输入以下内容：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">blog.views.article</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">blog.views.common</span> <span class="ss">:as</span> <span class="nv">common</span><span class="p">])</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:use</span> <span class="p">[</span><span class="nv">noir.core</span><span class="p">]</span>
</span><span class='line'>        <span class="p">[</span><span class="nv">hiccup.form-helpers</span><span class="p">]))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="nf">defpage</span> <span class="s">&quot;/blog/add&quot;</span> <span class="p">[]</span>
</span><span class='line'>  <span class="p">(</span><span class="nf">common/layout</span>
</span><span class='line'>    <span class="p">[</span><span class="ss">:h1</span> <span class="s">&quot;新建文章&quot;</span><span class="p">]</span>
</span><span class='line'>    <span class="p">(</span><span class="nf">form-to</span> <span class="p">[</span><span class="ss">:post</span> <span class="s">&quot;/blog/add&quot;</span><span class="p">]</span>
</span><span class='line'>             <span class="p">(</span><span class="nf">label</span> <span class="s">&quot;title&quot;</span> <span class="s">&quot;标题：&quot;</span><span class="p">)</span>
</span><span class='line'>             <span class="p">(</span><span class="nf">text-field</span> <span class="p">{</span><span class="ss">:size</span> <span class="mi">50</span><span class="p">}</span> <span class="s">&quot;title&quot;</span><span class="p">)</span> <span class="p">[</span><span class="ss">:br</span><span class="p">]</span>
</span><span class='line'>             <span class="p">(</span><span class="nf">label</span> <span class="s">&quot;content&quot;</span> <span class="s">&quot;内容：&quot;</span><span class="p">)</span>
</span><span class='line'>             <span class="p">(</span><span class="nf">text-area</span> <span class="p">{</span><span class="ss">:rows</span> <span class="mi">20</span> <span class="ss">:cols</span> <span class="mi">50</span><span class="p">}</span> <span class="s">&quot;content&quot;</span><span class="p">)</span> <span class="p">[</span><span class="ss">:br</span><span class="p">]</span>
</span><span class='line'>             <span class="p">(</span><span class="nf">submit-button</span> <span class="s">&quot;提交&quot;</span><span class="p">))))</span>
</span></code></pre></td></tr></table></div></figure>


<p><code>defpage</code>和<code>common/layout</code>我们之前已经见到过，前者定义了URL<code>/blog/add</code>指向的页面，后者则是套用了一个模板。<code>[:h1 ...]</code>和<code>[:br]</code>也应该熟悉，它们是Hiccup的语法，分别生成<code>&lt;h1&gt;...&lt;/h1&gt;</code>和<code>&lt;br&gt;</code>标签。</p>

<!-- more -->


<p><code>form-to</code>是一个新的语法，不过从名字上也可以猜到，它用来生成一个<code>&lt;form&gt;</code>标签，结合<code>[:post "/blog/add"]</code>一起来看就是<code>&lt;form action="/blog/add" method="post"&gt;...&lt;/form&gt;</code>。至于<code>label</code>、<code>text-field</code>、<code>text-area</code>、以及<code>submit-button</code>都是用来生成相应的表单标签的，它们包含在hiccup.form-helpers命名空间中，具体用法可以到REPL中查看它们的文档，如：（在项目目录中执行<code>lein repl</code>）</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="nv">blog.server=&gt;</span> <span class="p">(</span><span class="nf">use</span> <span class="ss">&#39;hiccup.form-helpers</span><span class="p">)</span>
</span><span class='line'><span class="nv">nil</span>
</span><span class='line'><span class="nv">blog.server=&gt;</span> <span class="p">(</span><span class="nb">doc </span><span class="nv">label</span><span class="p">)</span>
</span><span class='line'><span class="nv">-------------------------</span>
</span><span class='line'><span class="nv">hiccup.form-helpers/label</span>
</span><span class='line'><span class="p">([</span><span class="nb">name </span><span class="nv">text</span><span class="p">])</span> <span class="c1">; 这个函数接受两个参数，第一个参数是for属性，第二个是它的文本，即&lt;label for=&quot;name&quot;&gt;text&lt;/label&gt;。</span>
</span><span class='line'>  <span class="nv">Creates</span> <span class="nv">a</span> <span class="nv">label</span> <span class="nb">for </span><span class="nv">an</span> <span class="nv">input</span> <span class="nv">field</span> <span class="nv">with</span> <span class="nv">the</span> <span class="nv">supplied</span> <span class="nv">name.</span>
</span><span class='line'><span class="nv">nil</span>
</span><span class='line'><span class="nv">blog.server=&gt;</span> <span class="p">(</span><span class="nb">doc </span><span class="nv">text-field</span><span class="p">)</span>
</span><span class='line'><span class="nv">-------------------------</span>
</span><span class='line'><span class="nv">hiccup.form-helpers/text-field</span>
</span><span class='line'><span class="p">([</span><span class="nv">name</span><span class="p">]</span> <span class="p">[</span><span class="nb">name </span><span class="nv">value</span><span class="p">])</span> <span class="c1">; 这个函数可以传一个或两个参数，第一个参数是name属性，第二个参数是value，即&lt;input type=&quot;text&quot; name=&quot;name&quot; value=&quot;value&quot;&gt;。</span>
</span><span class='line'>  <span class="nv">Creates</span> <span class="nv">a</span> <span class="k">new </span><span class="nv">text</span> <span class="nv">input</span> <span class="nv">field.</span>
</span><span class='line'><span class="nv">nil</span>
</span></code></pre></td></tr></table></div></figure>


<p><code>{:size 50}</code>是个很特别的地方，虽然从字面上就能猜出它是<code>&lt;input&gt;</code>标签的<code>size</code>属性，用来设置文本框的长度的，但为什么会是这样的语法呢？这是Clojure定义的吗？当然不是。还记得我们之前提过的宏吗？开发者可以用宏来定义新的语法，Hiccup就定义了这样的语法，可以用map的形式传入额外的HTML属性。尝试在REPL中执行<code>(html [:font {:color "red"} "hi"])</code>，看看结果是什么吧。</p>

<h3>接收表单信息</h3>

<p>接下来我们再创建一个页面来接收表单信息。Noir可以按照HTTP方式的不同（GET、POST、DELETE等）来进行路由，比如同样是<code>/blog/add</code>这个URL，我们可以为它创建一个独立的页面，响应POST请求：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="nf">defpage</span> <span class="p">[</span><span class="ss">:post</span> <span class="s">&quot;/blog/add&quot;</span><span class="p">]</span> <span class="p">[]</span>
</span><span class='line'>  <span class="s">&quot;添加成功&quot;</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>尝试提交刚才的页面，会发现得到了预期结果：添加成功。那如何接收表单信息呢？</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="nf">defpage</span> <span class="p">[</span><span class="ss">:post</span> <span class="s">&quot;/blog/add&quot;</span><span class="p">]</span> <span class="p">{</span><span class="ss">:as</span> <span class="nv">forms</span><span class="p">}</span>
</span><span class='line'>  <span class="p">(</span><span class="nb">str </span><span class="s">&quot;添加成功，文章标题是：&quot;</span> <span class="p">(</span><span class="ss">:title</span> <span class="nv">forms</span><span class="p">)))</span>
</span></code></pre></td></tr></table></div></figure>


<p>似乎又多了几个新奇的语法，我们一一来解释：</p>

<p><code>{:as forms}</code>是一种解构（destructuring）语法，解构的对象是list或map，将它们包含的元素拆解出来。Noir在调用页面函数时（defpage实质上是创建了一个函数）会将接收到的参数以map的形式传递给该函数，如<code>title=greeting&amp;content=helloworld</code>会以`{:title &#8220;greeting&#8221;, :content &#8220;helloworld&#8221;}的形式传递过来，函数可以通过以下几种方式对map类型进行解构：</p>

<ul>
<li>不接收参数，使用<code>[]</code>来表示。</li>
<li>接收指定名称的参数，如<code>{title :title, content :content}</code>，它会将map中键名为:title的值赋给title变量，:content的内容赋给content变量，其他的键名会丢弃。如果键名很多，可以用这种缩写形式：<code>{:keys [title content]}</code>。</li>
<li>接收整个map，使用<code>{:as forms}</code>，其中forms是自定义的，这样就能从forms变量中获取某个键的值。</li>
<li>将以上两者结合，即<code>{title :title, content :content, :as forms}</code>，需要注意的是forms中还是包含:title和:content的，不会因为它们已经被赋值给其他变量了而从map中剔除掉。</li>
</ul>


<p>你可以将上面这段代码中的<code>{:as forms}</code>替换成其他形式来进行实验，看看是否真的掌握了解构的用法。至于对list对象的解构，我们以后会遇到。</p>

<p>如何获取map中某个键的值？之前我们在与Java交互时提过有两种方法：<code>(get forms :title)</code>和<code>(.get forms :title)</code>，这里展示的是第三种：<code>(:title forms)</code>，即用关键字作为一个函数，获取map中的值。如果键不存在则返回nil，可以提供默认值：<code>(:title forms "Default Title")</code>。</p>

<p><code>str</code>则是一个函数，会将它所接收到的所有参数转换成字符串并拼接起来（中间不会添加空格）。</p>

<h3>表单验证</h3>

<p>“永远不要相信用户输入的信息”，我们必须对表单内容进行验证，比如标题为空时我们应该显示错误信息，并让用户重新填写。Noir提供了表单验证的相关函数，位于noir.validation命名空间下。下面我们就来添加简单的验证功能：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">...</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">noir.validation</span> <span class="ss">:as</span> <span class="nv">vali</span><span class="p">]))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="kd">defn </span><span class="nv">valid?</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">title</span> <span class="nv">content</span><span class="p">]}]</span>
</span><span class='line'>  <span class="p">(</span><span class="nf">vali/rule</span> <span class="p">(</span><span class="nf">vali/has-value?</span> <span class="nv">title</span><span class="p">)</span>
</span><span class='line'>             <span class="p">[</span><span class="ss">:title</span> <span class="s">&quot;标题不能为空。&quot;</span><span class="p">])</span>
</span><span class='line'>  <span class="p">(</span><span class="nf">vali/rule</span> <span class="p">(</span><span class="nf">vali/min-length?</span> <span class="nv">title</span> <span class="mi">10</span><span class="p">)</span>
</span><span class='line'>             <span class="p">[</span><span class="ss">:title</span> <span class="s">&quot;标题不能少于10个字符。&quot;</span><span class="p">])</span>
</span><span class='line'>  <span class="p">(</span><span class="nf">vali/rule</span> <span class="p">(</span><span class="nf">vali/has-value?</span> <span class="nv">content</span><span class="p">)</span>
</span><span class='line'>             <span class="p">[</span><span class="ss">:content</span> <span class="s">&quot;内容不能为空。&quot;</span><span class="p">])</span>
</span><span class='line'>  <span class="p">(</span><span class="nb">not </span><span class="p">(</span><span class="nf">vali/errors?</span> <span class="ss">:title</span> <span class="ss">:content</span><span class="p">)))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="nf">defpage</span> <span class="p">[</span><span class="ss">:post</span> <span class="s">&quot;/blog/add&quot;</span><span class="p">]</span> <span class="p">{</span><span class="ss">:as</span> <span class="nv">forms</span><span class="p">}</span>
</span><span class='line'>  <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">valid?</span> <span class="nv">forms</span><span class="p">)</span>
</span><span class='line'>    <span class="p">(</span><span class="nb">str </span><span class="s">&quot;添加成功，文章标题是：&quot;</span> <span class="p">(</span><span class="ss">:title</span> <span class="nv">forms</span><span class="p">))</span>
</span><span class='line'>    <span class="p">(</span><span class="nb">str </span><span class="p">(</span><span class="nf">vali/get-errors</span> <span class="ss">:title</span><span class="p">)</span> <span class="p">(</span><span class="nf">vali/get-errors</span> <span class="ss">:content</span><span class="p">))))</span>
</span></code></pre></td></tr></table></div></figure>


<p>这段代码的运行效果是：如果提交的表单中标题和内容都有值，则显示“添加成功”，否则提示“标题不为空”、“内容不为空”等。我们来分析一下这段代码。</p>

<p><code>defn</code>定义一个函数，它的参数使用了前面提到的解构，函数体则是由三条语句构成。<code>valid?</code>是合法的函数名吗？前面提过Clojure的变量可以包含特殊字符，所以函数名中是可以存在<code>?</code>、<code>!</code>等字符的。当然我们也有一些习惯，比如以<code>?</code>结尾的函数名一般会返回布尔型。</p>

<p><code>vali/rule</code>函数用来描述一个验证规则，它的第一个参数是一个能够返回布尔型的表达式，第二个参数是一个向量（vector），包含两个元素，分别是字段名和错误提示信息，用于生成一个包含所有错误信息的map。以上面这段代码为例，如果三条验证都不通过，那生成的错误信息会是<code>{:title ["标题不能为空。" "标题不能少于10个字符。"], :content ["内容不能为空。"]}</code>。不过，这个map是由<code>noir.validation</code>维护的，我们不能直接获取到。</p>

<p><code>vali/errors?</code>接收一个字段列表，如果有一个字段验证不通过（产生了错误信息）则返回真，<code>not</code>函数自然就是将这个“真”转换为“假”，从而和<code>valid?</code>的语义一致，即不合法（验证不通过）。</p>

<p>最后，<code>vali/get-errors</code>函数可以将验证过程中生成的错误信息按照字段名提取出来。</p>

<p>这里我们还第一次遇到了流程控制语句：<code>if</code>，它和<code>let</code>一样是一种“特殊形式(Special Form)”。它的一般格式是<code>(if 布尔型 语句1 语句2)</code>，如<code>(if (&gt; 1 2) (println true) (println false))</code>。如果语句包含多行怎么办？可以使用<code>do</code>函数。如果条件分支只有一个，则可以使用<code>when</code>和<code>when-not</code>，这时可以直接包含多行语句，不需要使用<code>do</code>。以下是一些示例：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nb">&gt; </span><span class="mi">1</span> <span class="mi">2</span><span class="p">)</span>
</span><span class='line'>  <span class="p">(</span><span class="nf">do</span>
</span><span class='line'>    <span class="p">(</span><span class="nb">println </span><span class="mi">1</span><span class="p">)</span>
</span><span class='line'>    <span class="p">(</span><span class="nb">println </span><span class="mi">2</span><span class="p">))</span>
</span><span class='line'>  <span class="p">(</span><span class="nb">println </span><span class="mi">3</span><span class="p">))</span> <span class="c1">;-&gt; 3</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="nb">when </span><span class="p">(</span><span class="nb">&lt; </span><span class="mi">1</span> <span class="mi">2</span><span class="p">)</span>
</span><span class='line'>  <span class="p">(</span><span class="nb">println </span><span class="mi">1</span><span class="p">)</span>
</span><span class='line'>  <span class="p">(</span><span class="nb">println </span><span class="mi">2</span><span class="p">))</span> <span class="c1">;-&gt; 1 \n 2</span>
</span></code></pre></td></tr></table></div></figure>


<h3>错误提示</h3>

<p>那如何实现这样的需求：如果表单验证不通过，则重新显示表单，加入用户之前提交的内容，并显示出错信息。要做到这一点，就需要使用两个新的函数，<code>vali/on-error</code>和<code>noir.core/render</code>，并对<code>/blog/add</code>页面做一些修改。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="nf">defpartial</span> <span class="nv">error-item</span> <span class="p">[[</span><span class="nv">first-error</span><span class="p">]]</span>
</span><span class='line'>  <span class="p">[</span><span class="ss">:span.error</span> <span class="nv">first-error</span><span class="p">])</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="nf">defpage</span> <span class="s">&quot;/blog/add&quot;</span> <span class="p">{</span><span class="ss">:as</span> <span class="nv">forms</span><span class="p">}</span>
</span><span class='line'>  <span class="p">(</span><span class="nf">common/layout</span>
</span><span class='line'>    <span class="p">[</span><span class="ss">:h1</span> <span class="s">&quot;新建文章&quot;</span><span class="p">]</span>
</span><span class='line'>    <span class="p">(</span><span class="nf">form-to</span> <span class="p">[</span><span class="ss">:post</span> <span class="s">&quot;/blog/add&quot;</span><span class="p">]</span>
</span><span class='line'>             <span class="p">(</span><span class="nf">label</span> <span class="s">&quot;title&quot;</span> <span class="s">&quot;标题：&quot;</span><span class="p">)</span>
</span><span class='line'>             <span class="p">(</span><span class="nf">text-field</span> <span class="p">{</span><span class="ss">:size</span> <span class="mi">50</span><span class="p">}</span> <span class="s">&quot;title&quot;</span> <span class="p">(</span><span class="ss">:title</span> <span class="nv">forms</span><span class="p">))</span>
</span><span class='line'>             <span class="p">(</span><span class="nf">vali/on-error</span> <span class="ss">:title</span> <span class="nv">error-item</span><span class="p">)</span> <span class="p">[</span><span class="ss">:br</span><span class="p">]</span>
</span><span class='line'>             <span class="p">(</span><span class="nf">label</span> <span class="s">&quot;content&quot;</span> <span class="s">&quot;内容：&quot;</span><span class="p">)</span>
</span><span class='line'>             <span class="p">(</span><span class="nf">text-area</span> <span class="p">{</span><span class="ss">:rows</span> <span class="mi">20</span> <span class="ss">:cols</span> <span class="mi">50</span><span class="p">}</span> <span class="s">&quot;content&quot;</span> <span class="p">(</span><span class="ss">:content</span> <span class="nv">forms</span><span class="p">))</span>
</span><span class='line'>             <span class="p">(</span><span class="nf">vali/on-error</span> <span class="ss">:content</span> <span class="nv">error-item</span><span class="p">)</span> <span class="p">[</span><span class="ss">:br</span><span class="p">]</span>
</span><span class='line'>             <span class="p">(</span><span class="nf">submit-button</span> <span class="s">&quot;提交&quot;</span><span class="p">))))</span>
</span><span class='line'>
</span><span class='line'><span class="c1">; 此处省略valid?函数，它没有变化</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="nf">defpage</span> <span class="p">[</span><span class="ss">:post</span> <span class="s">&quot;/blog/add&quot;</span><span class="p">]</span> <span class="p">{</span><span class="ss">:as</span> <span class="nv">forms</span><span class="p">}</span>
</span><span class='line'>  <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">valid?</span> <span class="nv">forms</span><span class="p">)</span>
</span><span class='line'>    <span class="p">(</span><span class="nb">str </span><span class="s">&quot;添加成功，文章标题是：&quot;</span> <span class="p">(</span><span class="ss">:title</span> <span class="nv">forms</span><span class="p">))</span>
</span><span class='line'>    <span class="p">(</span><span class="nf">render</span> <span class="s">&quot;/blog/add&quot;</span> <span class="nv">forms</span><span class="p">)))</span>
</span></code></pre></td></tr></table></div></figure>


<p>先看<code>render</code>，它是Noir提供的一个函数，能够在页面中渲染另一个页面的内容，就像调用一个函数一样。这里我们则是在表单提交的页面里渲染了“新建文章”页面的内容，并将表单参数传递了过去。</p>

<p>对“新建文章”页面我们做了以下修改：</p>

<ul>
<li>接收参数，并作为<code>forms</code>变量保存这个map。</li>
<li>为<code>text-field</code>和<code>text-area</code>两个表单控件添加了默认值。</li>
<li>调用了<code>vali/on-error</code>函数，当某个字段包含错误信息时，它会调用第二个参数所指向的函数（这里是<code>error-item</code>），并将该字段的错误信息作为参数传递给这个函数。</li>
</ul>


<p><code>error-item</code>函数的功能很简单，将接受到的错误信息渲染成一个<code>&lt;span&gt;</code>标签展示出来。这里的<code>[:span.error ...]</code>会被解析成<code>&lt;span class="error"&gt;...&lt;/span&gt;</code>。至于<code>[[first-error]]</code>，它是一种对list对象的解构操作。前面我们看到在错误信息中，某个字段即使只有一条错误信息，也会以向量的形式保存。我们这里只需要每个字段的第一条错误信息，所以使用了这种形式。你可以这样重写<code>error-item</code>，效果是一样的：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="nf">defpartial</span> <span class="nv">error-item</span> <span class="p">[</span><span class="nv">errors</span><span class="p">]</span>
</span><span class='line'>  <span class="p">[</span><span class="ss">:span.error</span> <span class="p">(</span><span class="nb">first </span><span class="nv">errors</span><span class="p">)])</span>
</span></code></pre></td></tr></table></div></figure>


<h2>操作数据库</h2>

<p>Clojure程序连接数据库可以使用<code>clojure.java.jdbc</code>这个类库，它能够操作MySQL、PostgreSQL、SQLite、MSSQL等。这里我们将演示如何连接MySQL数据库，因此除了<code>clojure.java.jdbc</code>外，还需要添加MySQL Connector依赖项：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">defproject </span><span class="nv">...</span>
</span><span class='line'>            <span class="ss">:dependencies</span> <span class="p">[</span><span class="nv">...</span>
</span><span class='line'>                           <span class="p">[</span><span class="nv">org.clojure/java.jdbc</span> <span class="s">&quot;0.2.3&quot;</span><span class="p">]</span>
</span><span class='line'>                           <span class="p">[</span><span class="nv">mysql/mysql-connector-java</span> <span class="s">&quot;5.1.6&quot;</span><span class="p">]]</span>
</span><span class='line'>            <span class="nv">...</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>为了保存博客文章，我们在本地MySQL服务中新建一个<code>blog_db</code>数据库，并赋予用户<code>blog_db</code>（密码相同）该库的所有权限。然后我们会建立一张article表，用于保存文章：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class='mysql'><span class='line'><span class="err">$</span> <span class="n">mysql</span> <span class="o">-</span><span class="n">uroot</span> <span class="o">-</span><span class="n">p</span>
</span><span class='line'><span class="n">mysql</span><span class="o">&gt;</span> <span class="k">create</span> <span class="k">database</span> <span class="n">blog_db</span> <span class="k">collate</span> <span class="n">utf8_general_ci</span><span class="p">;</span>
</span><span class='line'><span class="n">mysql</span><span class="o">&gt;</span> <span class="k">grant</span> <span class="k">all</span> <span class="k">on</span> <span class="n">blog_db</span><span class="p">.</span><span class="o">*</span> <span class="k">to</span> <span class="s1">&#39;blog_db&#39;</span><span class="o">@</span><span class="s1">&#39;localhost&#39;</span> <span class="n">identified</span> <span class="k">by</span> <span class="s1">&#39;blog_db&#39;</span><span class="p">;</span>
</span><span class='line'><span class="n">mysql</span><span class="o">&gt;</span> <span class="k">CREATE</span> <span class="k">TABLE</span> <span class="ss">`article`</span> <span class="p">(</span>
</span><span class='line'>    <span class="o">-&gt;</span>   <span class="ss">`id`</span> <span class="kt">int</span><span class="p">(</span><span class="mi">11</span><span class="p">)</span> <span class="k">NOT</span> <span class="no">NULL</span> <span class="kp">AUTO_INCREMENT</span><span class="p">,</span>
</span><span class='line'>    <span class="o">-&gt;</span>   <span class="ss">`title`</span> <span class="kt">varchar</span><span class="p">(</span><span class="mi">500</span><span class="p">)</span> <span class="k">NOT</span> <span class="no">NULL</span><span class="p">,</span>
</span><span class='line'>    <span class="o">-&gt;</span>   <span class="ss">`content`</span> <span class="kt">text</span> <span class="k">NOT</span> <span class="no">NULL</span><span class="p">,</span>
</span><span class='line'>    <span class="o">-&gt;</span>   <span class="ss">`user_id`</span> <span class="kt">int</span><span class="p">(</span><span class="mi">11</span><span class="p">)</span> <span class="k">NOT</span> <span class="no">NULL</span><span class="p">,</span>
</span><span class='line'>    <span class="o">-&gt;</span>   <span class="ss">`created`</span> <span class="kt">datetime</span> <span class="k">NOT</span> <span class="no">NULL</span><span class="p">,</span>
</span><span class='line'>    <span class="o">-&gt;</span>   <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="p">(</span><span class="ss">`id`</span><span class="p">)</span>
</span><span class='line'>    <span class="o">-&gt;</span> <span class="p">)</span> <span class="kp">ENGINE</span><span class="o">=</span><span class="n">InnoDB</span><span class="p">;</span>
</span></code></pre></td></tr></table></div></figure>


<p>我们新建一个<code>src/blog/database.clj</code>文件，用来存放数据库连接信息：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">blog.database</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="k">def </span><span class="nv">db-spec</span> <span class="p">{</span><span class="ss">:classname</span> <span class="s">&quot;com.mysql.jdbc.Driver&quot;</span>
</span><span class='line'>              <span class="ss">:subprotocol</span> <span class="s">&quot;mysql&quot;</span>
</span><span class='line'>              <span class="ss">:subname</span> <span class="s">&quot;//127.0.0.1:3306/blog_db?useUnicode=true&amp;characterEncoding=UTF-8&quot;</span>
</span><span class='line'>              <span class="ss">:user</span> <span class="s">&quot;blog_db&quot;</span>
</span><span class='line'>              <span class="ss">:password</span> <span class="s">&quot;blog_db&quot;</span><span class="p">})</span>
</span></code></pre></td></tr></table></div></figure>


<p>接着新建<code>src/blog/models/blog.clj</code>文件，编写插入记录的函数：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">blog.models.blog</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">clojure.java.jdbc</span> <span class="ss">:as</span> <span class="nv">sql</span><span class="p">])</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:use</span> <span class="p">[</span><span class="nv">blog.database</span> <span class="ss">:only</span> <span class="p">[</span><span class="nv">db-spec</span><span class="p">]])</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:import</span> <span class="nv">java.sql.Timestamp</span><span class="p">))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="kd">defn </span><span class="nv">add!</span> <span class="p">[</span><span class="nv">forms</span><span class="p">]</span>
</span><span class='line'>  <span class="p">(</span><span class="nf">sql/with-connection</span> <span class="nv">db-spec</span>
</span><span class='line'>    <span class="p">(</span><span class="nf">sql/insert-records</span> <span class="s">&quot;article&quot;</span>
</span><span class='line'>      <span class="p">{</span><span class="ss">:title</span> <span class="p">(</span><span class="ss">:title</span> <span class="nv">forms</span><span class="p">)</span>
</span><span class='line'>       <span class="ss">:content</span> <span class="p">(</span><span class="ss">:content</span> <span class="nv">forms</span><span class="p">)</span>
</span><span class='line'>       <span class="ss">:user_id</span> <span class="mi">0</span>
</span><span class='line'>       <span class="ss">:created</span> <span class="p">(</span><span class="nf">Timestamp.</span> <span class="p">(</span><span class="nf">System/currentTimeMillis</span><span class="p">))})))</span>
</span></code></pre></td></tr></table></div></figure>


<p>最后，我们只需要在<code>src/blog/views/article.clj</code>中调用这个函数即可：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">...</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:require</span> <span class="nv">...</span>
</span><span class='line'>            <span class="p">[</span><span class="nv">blog.models.blog</span> <span class="ss">:as</span> <span class="nv">model-blog</span><span class="p">]))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="nf">defpage</span> <span class="p">[</span><span class="ss">:post</span> <span class="s">&quot;/blog/add&quot;</span><span class="p">]</span> <span class="p">{</span><span class="ss">:as</span> <span class="nv">forms</span><span class="p">}</span>
</span><span class='line'>  <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">valid?</span> <span class="nv">forms</span><span class="p">)</span>
</span><span class='line'>    <span class="p">(</span><span class="nb">str </span><span class="s">&quot;添加成功，文章编号是：&quot;</span>
</span><span class='line'>         <span class="p">(</span><span class="ss">:generated_key</span> <span class="p">(</span><span class="nb">first </span><span class="p">(</span><span class="nf">model-blog/add!</span> <span class="nv">forms</span><span class="p">))))</span>
</span><span class='line'>    <span class="p">(</span><span class="nf">render</span> <span class="s">&quot;/blog/add&quot;</span> <span class="nv">forms</span><span class="p">)))</span>
</span></code></pre></td></tr></table></div></figure>


<p>这里我们见到了<code>clojure.java.jdbc</code>命名空间下的两个函数：<code>with-connection</code>和<code>insert-records</code>。前者用来打开一个数据库连接，它的第一个参数可以是一个map，表示数据库的连接信息，也可以是字符串，还可以直接传递一个DataSource对象，这点我们会在如何使用连接池时讲解。当<code>with-connection</code>执行完毕时，数据库连接也会随之关闭。<code>insert-record</code>则用于插入一条或多条数据，第一个参数是数据表名，第二个参数开始则是将要插入的记录。这个函数的返回值你应该可以从<code>(:generated_key ...)</code>这段代码中猜出来。</p>

<p>注意，我们的数据表中有一个DATETIME类型的字段，它需要使用<code>java.sql.Timestamp</code>类型来赋值。Clojure中引入一个类可以使用<code>import</code>函数，<code>ns</code>宏提供了便捷的方式<code>:import</code>。当需要一次导入多个类时，可以使用<code>(:import (java.sql Timestamp Date Time))</code>。</p>

<h3>使用C3P0连接池</h3>

<p>在高性能网站中，频繁开关数据库连接不是个好主意，通常的方式是使用连接池。这里我们演示如何使用C3P0连接池。</p>

<ul>
<li>添加依赖项：<code>[c3p0 "0.9.1.2"]</code></li>
<li>修改<code>src/blog/database.clj</code>，添加以下代码：</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">blog.database</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:import</span> <span class="nv">com.mchange.v2.c3p0.ComboPooledDataSource</span><span class="p">))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="k">def </span><span class="nv">db-spec</span> <span class="nv">...</span><span class="p">)</span> <span class="c1">; db-spec没有变化，此处省略</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="k">def </span><span class="nv">pool</span>
</span><span class='line'>  <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nv">cpds</span> <span class="p">(</span><span class="nb">doto </span><span class="p">(</span><span class="nf">ComboPooledDataSource.</span><span class="p">)</span>
</span><span class='line'>               <span class="p">(</span><span class="nf">.setDriverClass</span> <span class="p">(</span><span class="ss">:classname</span> <span class="nv">db-spec</span><span class="p">))</span>
</span><span class='line'>               <span class="p">(</span><span class="nf">.setJdbcUrl</span> <span class="p">(</span><span class="nb">str </span><span class="s">&quot;jdbc:&quot;</span> <span class="p">(</span><span class="ss">:subprotocol</span> <span class="nv">spec</span><span class="p">)</span> <span class="s">&quot;:&quot;</span> <span class="p">(</span><span class="ss">:subname</span> <span class="nv">db-spec</span><span class="p">)))</span>
</span><span class='line'>               <span class="p">(</span><span class="nf">.setUser</span> <span class="p">(</span><span class="ss">:user</span> <span class="nv">db-spec</span><span class="p">))</span>
</span><span class='line'>               <span class="p">(</span><span class="nf">.setPassword</span> <span class="p">(</span><span class="ss">:password</span> <span class="nv">db-spec</span><span class="p">)))]</span>
</span><span class='line'>    <span class="p">{</span><span class="ss">:datasource</span> <span class="nv">cpds</span><span class="p">}))</span>
</span></code></pre></td></tr></table></div></figure>


<ul>
<li>更换<code>src/blog/models/blog.clj</code>中<code>with-connection</code>的参数：</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">...</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:use</span> <span class="p">[</span><span class="nv">blog.database</span> <span class="ss">:only</span> <span class="p">[</span><span class="nv">pool</span><span class="p">]]))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="kd">defn </span><span class="nv">add!</span> <span class="p">[</span><span class="nv">forms</span><span class="p">]</span>
</span><span class='line'>  <span class="p">(</span><span class="nf">sql/with-connection</span> <span class="nv">pool</span>
</span><span class='line'>    <span class="nv">...</span><span class="p">))</span>
</span></code></pre></td></tr></table></div></figure>


<p><code>with-connection</code>接收到的参数形式是<code>{:datasource DataSource接口的实例}</code>，<code>(ComboPooledDataSource.)</code>就是这样的一个实例，这种语法和<code>(new ComboPooledDataSource)</code>等价。<code>doto</code>函数表示连续调用第一个参数所指向的对象的方法，最后返回这个对象。这段代码可以有不同的写法，如<code>(def pool {:datasource (doto (Combo...))})</code>。</p>

<h2>小结</h2>

<p>这一章我们学习了如何配置Eclipse以编写Leiningen项目；如何使用表单和接受参数，特别是表单验证和错误信息的提示；最后我们演示了如何将数据保存到MySQL中，并使用连接池来优化项目。下一节我们将为博客增加用户登录的功能，以讲解Cookie和Session的使用。我们还会学习如何对Leiningen项目进行打包和发布，并尝试将我们的博客发布到PaaS平台Heroku上去。</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Clojure实战(1)：使用Noir框架开发博客(上)]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/11/25/cia-noir-1/"/>
    <updated>2012-11-25T16:58:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/11/25/cia-noir-1</id>
    <content type="html"><![CDATA[<h2>前言</h2>

<p>为什么要学习一门新的语言？我的想法很简单，平时OO、PO代码写多了，却从未接触过函数式编程，不免有些遗憾。考察下来，Clojure可以用来尝尝鲜，所以就决定学一学。为了给自己的学习留下些记录，就有了这样一份教程。</p>

<p>Clojure已经有一些不错的教程，如<a href="http://java.ociweb.com/mark/">Mark Volkmann</a>的<a href="http://java.ociweb.com/mark/clojure/article.html">Clojure - Functional Programming for the JVM</a>，Storm的主要贡献者<a href="http://xumingming.sinaapp.com/">徐明明</a>也对这个教程做了<a href="http://xumingming.sinaapp.com/302/clojure-functional-programming-for-the-jvm-clojure-tutorial/">全文翻译</a>。还有一些不错的书籍，像<a href="http://oreilly.com/">O&#8217;Reilly</a>的<a href="http://www.amazon.com/Clojure-Programming-Chas-Emerick/dp/1449394701">Clojure Programming</a>，都值得一读。我是从Mark的教程开始学起的，对其中没有提到的部分则是参考了Clojure Programming这本书。Clojure的<a href="http://clojure.org/">官方网站</a>上有详尽的<a href="http://clojure.github.com/clojure/">API参考</a>，可以作为工具书查阅。</p>

<p>但是，上面提到的教程都是针对Clojure语言本身的，从 <em>Hello, world!</em> 开始，讲解Clojure的各种语法，关键字，结构等等。虽然Clojure的语法已经足够吸引你的眼球，在<a href="http://en.wikipedia.org/wiki/Read-eval-print_loop">REPL</a>中敲击Clojure代码已经是一种莫大的乐趣了，但似乎还有些不够，我们想看到一个用Clojure编写的应用程序！</p>

<p>因为平时都是做Web开发，所以先从一个Web框架入手会是不错的选择，因此这份教程会从使用<a href="http://webnoir.org/">Noir</a>框架搭建一个博客开始，带你领略Clojure的魅力。</p>

<!-- more -->


<h2>一句话概述Clojure</h2>

<p><strong>Clojure是一种运行在JVM平台上的函数式编程语言。</strong></p>

<ul>
<li><a href="http://en.wikipedia.org/wiki/Jvm">JVM平台</a>：历史悠久，应用广泛，成熟稳定。Clojure可以和Java程序交互，调用各种类库，与现有系统整合。</li>
<li><a href="http://en.wikipedia.org/wiki/Functional_programming">函数式编程</a>：<a href="http://en.wikipedia.org/wiki/LISP">Lisp</a>的一种方言，表达力强，是解决高并发问题的利器。</li>
</ul>


<h2>安装Clojure</h2>

<p>Clojure是以一个Jar包发行的，可以到<a href="http://clojure.org/getting_started">官网下载</a>后使用<code>java -jar</code>命令运行。而在实际开发中，我们会选择使用<a href="https://github.com/technomancy/leiningen">Leiningen</a>或<a href="http://maven.apache.org/">Maven</a>来管理Clojure项目，本教程将以Leiningen（命令行是<code>lein</code>）作为项目管理工具进行讲解。</p>

<h3>安装Leiningen</h3>

<p><code>lein</code>目前有1.x和2.x两个版本，后者还在alpha阶段。使用以下命令安装<code>lein</code> 1.x版本：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span><span class="nb">cd</span> ~/bin <span class="c"># 假设$HOME/bin目录在系统的$PATH中</span>
</span><span class='line'><span class="nv">$ </span>wget https://raw.github.com/technomancy/leiningen/stable/bin/lein
</span><span class='line'><span class="nv">$ </span>chmod 755 lein
</span><span class='line'><span class="nv">$ </span>lein self-install
</span><span class='line'><span class="nv">$ </span>lein repl
</span><span class='line'>REPL started; server listening on localhost port 1096
</span><span class='line'><span class="nv">user</span><span class="o">=</span>&gt;
</span></code></pre></td></tr></table></div></figure>


<p>这样就已经安装好了<code>lein</code>和Clojure环境，并启动了一个REPL，可以直接运行Clojure代码：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="nb">+ </span><span class="mi">1</span> <span class="mi">2</span><span class="p">)</span>
</span><span class='line'><span class="mi">3</span>
</span><span class='line'><span class="nv">user=&gt;</span>
</span></code></pre></td></tr></table></div></figure>


<p>这里出现了Clojure的两个特点：圆括号和前缀表达式。Clojure的基本语法是<code>(fn1 arg1 (fn2 arg2 arg3))</code>。函数是Clojure中的“一等公民”，它即是可执行的代码，又是一种数据（类似闭包的概念）。以后我们会慢慢熟悉。</p>

<h3>新建项目</h3>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>lein new proj
</span><span class='line'><span class="nv">$ </span>find proj
</span><span class='line'>proj
</span><span class='line'>proj/project.clj
</span><span class='line'>proj/src/proj/core.clj
</span><span class='line'><span class="nv">$ </span>cat proj/project.clj
</span><span class='line'><span class="o">(</span>defproject proj <span class="s2">&quot;1.0.0-SNAPSHOT&quot;</span>
</span><span class='line'>  :description <span class="s2">&quot;FIXME: write description&quot;</span>
</span><span class='line'>  :dependencies <span class="o">[[</span>org.clojure/clojure <span class="s2">&quot;1.3.0&quot;</span><span class="o">]])</span>
</span></code></pre></td></tr></table></div></figure>


<p><code>lein new</code>命令用来创建一个Clojure项目骨架，最重要的文件是<code>project.clj</code>，它声明了项目的基本属性以及依赖包。</p>

<p><code>lein plugin</code>命令可以用来管理lein的插件，我们可以通过安装lein-noir插件来生成基于Noir的项目骨架：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>lein plugin install lein-noir 1.2.1
</span><span class='line'><span class="nv">$ </span>lein noir new blog
</span><span class='line'><span class="nv">$ </span>find blog
</span><span class='line'>blog
</span><span class='line'>blog/project.clj
</span><span class='line'>blog/resources/public/css/reset.css
</span><span class='line'>blog/resources/public/img
</span><span class='line'>blog/resources/public/js
</span><span class='line'>blog/src/blog/models
</span><span class='line'>blog/src/blog/server.clj
</span><span class='line'>blog/src/blog/views/common.clj
</span><span class='line'>blog/src/blog/views/welcome.clj
</span></code></pre></td></tr></table></div></figure>


<p>我们可以直接运行这个项目：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span><span class="nb">cd </span>blog
</span><span class='line'><span class="nv">$ </span>lein run
</span><span class='line'>Starting server...
</span><span class='line'>2012-11-29 22:34:39.174:INFO::jetty-6.1.25
</span><span class='line'>2012-11-29 22:34:39.237:INFO::Started SocketConnector@0.0.0.0:8080
</span></code></pre></td></tr></table></div></figure>


<p>浏览http://localhost:8080，就能看到项目的页面了。</p>

<h2>Noir项目的基本结构</h2>

<h3>项目基本信息：project.clj</h3>

<p>Clojure文件都是以<code>.clj</code>为扩展名的。项目根目录下的project.clj文件包含了一些基本信息，我们逐一分析：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">defproject </span><span class="nv">blog</span> <span class="s">&quot;0.1.0-SNAPSHOT&quot;</span>
</span><span class='line'>            <span class="ss">:description</span> <span class="s">&quot;FIXME: write this!&quot;</span>
</span><span class='line'>            <span class="ss">:dependencies</span> <span class="p">[[</span><span class="nv">org.clojure/clojure</span> <span class="s">&quot;1.3.0&quot;</span><span class="p">]</span>
</span><span class='line'>                           <span class="p">[</span><span class="nv">noir</span> <span class="s">&quot;1.2.1&quot;</span><span class="p">]]</span>
</span><span class='line'>            <span class="ss">:main</span> <span class="nv">blog.server</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p><a href="http://clojuredocs.org/leiningen/leiningen.core/defproject">defproject</a>是Leiningen定义的一个<a href="http://clojure.org/macros">宏</a>，用来描述项目的基本信息。宏在Clojure中是一个很重要的语言特性，简单地说，开发者可以用宏创造出新的语法。</p>

<p>:description、:main等是一种直接量(literal)，我们称之为关键字(keyword)，通常以<code>:</code>开头，主要用来作为哈希表(map)中的键名，这里则用来表示项目的某种信息，从名称上应该很好理解。</p>

<p><code>[1 ["b", false]]</code>中的<code>[...]</code>表示一个向量(vector)，它的元素可以是任意类型，元素之间以空格或逗号分隔。这行代码也展示了Clojure中其他几种直接量：数值型、字符串、布尔型。</p>

<p>依赖项的描述也很直观，<code>[groupId/artifactId "version"]</code>。Clojure使用了和Maven相似的包命名方式，当groupId和artifactId相同时，可以进行简写，如<code>[noir "1.2.1"]</code>等价于<code>[noir/noir "1.2.1"]</code>，这也是Clojure鼓励的做法。对依赖项进行修改后，可以运行<code>lein deps</code>命令进行安装。lein会先从<a href="https://clojars.org">Clojars</a>上查找和下载，不存在时再到<a href="http://search.maven.org/">Maven中央仓库</a>中搜索。</p>

<p>最后，:main顾名思义指向的是程序入口，它配置的是一个命名空间，其中会包含一个<code>-main</code>方法（注意方法名中的<code>-</code>）。Leiningen项目的目录结构也是按照命名空间来的，这点和Java一致。</p>

<h3>入口文件：src/blog/server.clj</h3>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">blog.server</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">noir.server</span> <span class="ss">:as</span> <span class="nv">server</span><span class="p">]))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="nf">server/load-views</span> <span class="s">&quot;src/blog/views/&quot;</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="kd">defn </span><span class="nv">-main</span> <span class="p">[</span><span class="o">&amp;</span> <span class="nv">m</span><span class="p">]</span>
</span><span class='line'>  <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nv">mode</span> <span class="p">(</span><span class="nb">keyword </span><span class="p">(</span><span class="nb">or </span><span class="p">(</span><span class="nb">first </span><span class="nv">m</span><span class="p">)</span> <span class="ss">:dev</span><span class="p">))</span>
</span><span class='line'>        <span class="nv">port</span> <span class="p">(</span><span class="nf">Integer.</span> <span class="p">(</span><span class="nb">get </span><span class="p">(</span><span class="nf">System/getenv</span><span class="p">)</span> <span class="s">&quot;PORT&quot;</span> <span class="s">&quot;8080&quot;</span><span class="p">))]</span>
</span><span class='line'>    <span class="p">(</span><span class="nf">server/start</span> <span class="nv">port</span> <span class="p">{</span><span class="ss">:mode</span> <span class="nv">mode</span>
</span><span class='line'>                        <span class="ss">:ns</span> <span class="ss">&#39;blog</span><span class="p">})))</span>
</span></code></pre></td></tr></table></div></figure>


<p>ns宏用于定义当前的命名空间，:require表示导入其它命名空间，:as则是将为导入的命名空间设置别名。注意这些关键字及其用法都是在ns宏中定义的，这也是为什么说宏可以用来创造新的语法。</p>

<p>关于命名空间，你在执行<code>lein repl</code>的时候可能会注意到，当在blog项目下执行时，提示符是<code>blog.server=&gt;</code>，而在其他目录下执行时是<code>user=&gt;</code>，因为user是Clojure默认的命名空间，可以通过<code>(ns myspace)</code>来切换成<code>myspace=&gt;</code>。</p>

<p><code>server/load-views</code>表示调用<code>server</code>命名空间下的<code>load-views</code>函数，后面的<code>src/blog/views/</code>则是函数的参数。在大多数语言中，函数名称不能包含特殊字符，如<code>-</code>，但Clojure中的变量名、函数名、关键字等都可以包含诸如*、+、!、>这样的特殊字符。其中一些字符是有特定含义的，如关键字必须以<code>:</code>开头，以<code>::</code>开头的则表示仅在当前命名空间中有效。这些约定需要注意。</p>

<p>defn宏用来定义一个函数，基本用法是<code>(defn 函数名 [参数列表] 语句1 语句2)</code>。如果参数数量不定，可以使用这样的语法<code>[arg1 arg2 &amp; args]</code>，这样当函数接收四个参数时，后两个参数会作为一个向量赋值给args变量。</p>

<p>关于let，它是继“宏”和“函数”之后出现的第三个术语，“特殊形式”(Special Form)。我们暂时不去了解它们之间的关系，先来看看let的作用。设想这样一个场景，我们在调用一个函数之前会准备一些参数传送给它，这些参数仅在函数内部可见，函数处理完后就会销毁。let则是将参数准备和函数调用这两步整合了起来。它的语法是<code>(let [变量1 表达式1 变量2 表达式2] 语句1 语句2)</code>，举例来说，尝试在REPL中执行以下命令，思考一下结果是如何得出的：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nv">x</span> <span class="mi">1</span> <span class="nv">y</span> <span class="p">(</span><span class="nb">+ </span><span class="mi">2</span> <span class="mi">3</span><span class="p">)]</span> <span class="p">(</span><span class="nb">+ </span><span class="nv">x</span> <span class="nv">y</span><span class="p">))</span>
</span><span class='line'><span class="mi">6</span>
</span><span class='line'><span class="nv">user=&gt;</span>
</span></code></pre></td></tr></table></div></figure>


<p>入口文件中的表达式看起来有些复杂，但逐步拆解后就会明白：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="k">def </span><span class="nv">m</span> <span class="p">[])</span> <span class="c1">; 定义一个变量，它是一个空向量，正如不带参数调用-main函数时一样。Clojure中分号表示注释。</span>
</span><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="nb">first </span><span class="nv">m</span><span class="p">)</span> <span class="c1">; 获取向量（序列）的第一个元素，这里为空。</span>
</span><span class='line'><span class="nv">nil</span>
</span><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="nb">or </span><span class="p">(</span><span class="nb">first </span><span class="nv">m</span><span class="p">)</span> <span class="ss">:dev</span><span class="p">)</span> <span class="c1">; 从左往右执行参数，若结果不为空(nil)则停止执行，并返回该结果。</span>
</span><span class='line'><span class="ss">:dev</span>
</span><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="nb">keyword </span><span class="p">(</span><span class="nb">or </span><span class="p">(</span><span class="nb">first </span><span class="nv">m</span><span class="p">)</span> <span class="ss">:dev</span><span class="p">))</span> <span class="c1">; 获取关键字。由于传入的参数可能是一个关键字、一个变量、或一个字符串，因此使用keyword函数返回一个关键字类型。</span>
</span><span class='line'><span class="ss">:dev</span>
</span></code></pre></td></tr></table></div></figure>


<p>经过处理，mode变量包含了:dev这个关键字，且作用域仅在(let &#8230;)中有效。</p>

<p>再来看看port变量，这里涉及到了与Java类的交互：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="nf">System/getenv</span><span class="p">)</span> <span class="c1">; 这里的System不是一个命名空间，而是Java的一个类。通过这种方式我们调用了System类的静态方法getenv，并返回了一个Map类实例。</span>
</span><span class='line'><span class="nv">java.util.Collections$UnmodifiableMap</span>
</span><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="nb">get </span><span class="p">(</span><span class="nf">System/getenv</span><span class="p">)</span> <span class="s">&quot;PORT&quot;</span> <span class="s">&quot;8080&quot;</span><span class="p">)</span> <span class="c1">; 这里的get不是Map实例的get方法，而是Clojure中的一个函数，用于返回哈希表(map)中的值，不存在则返回一个默认值。</span>
</span><span class='line'><span class="s">&quot;8080&quot;</span>
</span><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="nf">.get</span> <span class="p">(</span><span class="nf">System/getenv</span><span class="p">)</span> <span class="s">&quot;PORT&quot;</span><span class="p">)</span> <span class="c1">; 这才是调用Map实例的get方法，注意点号和函数的参数。</span>
</span><span class='line'><span class="nv">nil</span>
</span><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="nf">Integer.</span> <span class="p">(</span><span class="nb">get </span><span class="p">(</span><span class="nf">System/getenv</span><span class="p">)</span> <span class="s">&quot;PORT&quot;</span> <span class="s">&quot;8080&quot;</span><span class="p">))</span> <span class="c1">; 又是一个和Java交互的语法：创建实例。它和以下语法等价：</span>
</span><span class='line'><span class="mi">8080</span>
</span><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="k">new </span><span class="nv">Integer</span> <span class="p">(</span><span class="nb">get </span><span class="p">(</span><span class="nf">System/getenv</span><span class="p">)</span> <span class="s">&quot;PORT&quot;</span> <span class="s">&quot;8080&quot;</span><span class="p">))</span>
</span><span class='line'><span class="mi">8080</span>
</span></code></pre></td></tr></table></div></figure>


<p>关于map再补充一点，它虽然是Clojure的一种数据类型，但底层其实是Map接口一个实现，因此以下语法是合法的。类似的情况在Clojure中还有很多。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="k">def </span><span class="nv">m</span> <span class="p">{</span><span class="ss">:a</span> <span class="mi">1</span>, <span class="ss">:b</span> <span class="mi">2</span><span class="p">})</span> <span class="c1">; map的语法是{键1 值1 键2 值2}，为了加强可读性，这里使用了逗号分隔了两组键值，Clojure在编译时会将逗号转换成空格。</span>
</span><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="nb">&lt; </span><span class="p">(</span><span class="nb">get </span><span class="nv">m</span> <span class="ss">:a</span><span class="p">)</span> <span class="p">(</span><span class="nf">.get</span> <span class="nv">m</span> <span class="ss">:b</span><span class="p">))</span> <span class="c1">; 1 &lt; 2</span>
</span><span class='line'><span class="nv">true</span>
</span></code></pre></td></tr></table></div></figure>


<p>参数的赋值就结束了，后面的代码也很好理解：调用server命名空间下的start函数，参数是监听端口和一组由map表示的参数。这里blog之前的单引号需要注意，表示其后的代码不需要进行解析(evaluate)，在表示命空间名时都需要加上（ns宏除外），如：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="nf">require</span> <span class="ss">&#39;noir.server</span><span class="p">)</span> <span class="c1">; 引入一个命名空间，使用noir.server/start调用方法。</span>
</span><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="nf">alias</span> <span class="ss">&#39;server</span> <span class="ss">&#39;noir.server</span><span class="p">)</span> <span class="c1">; 设置别名。</span>
</span><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="nb">refer </span><span class="ss">&#39;noir.server</span><span class="p">)</span> <span class="c1">; 将该命名空间下的变量导入当当前命名空间中，即可以直接使用(start ...)调用。</span>
</span><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="nf">use</span> <span class="ss">&#39;noir.server</span><span class="p">)</span> <span class="c1">; 同时完成require和refer。</span>
</span></code></pre></td></tr></table></div></figure>


<p><strong>小贴士</strong></p>

<p>这一节中我们引入了不少Clojure的函数、宏、特殊形式，有时会需要查阅这些函数的用法。除了上网查找API文档，还可以在REPL中使用<code>doc</code>和<code>source</code>函数来返回某个函数或宏的文档和源码：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="nv">user=&gt;</span> <span class="p">(</span><span class="nb">doc </span><span class="nv">first</span><span class="p">)</span>
</span><span class='line'><span class="nv">-------------------------</span>
</span><span class='line'><span class="nv">clojure.core/first</span>
</span><span class='line'><span class="p">([</span><span class="nv">coll</span><span class="p">])</span>
</span><span class='line'>  <span class="nv">Returns</span> <span class="nv">the</span> <span class="nb">first </span><span class="nv">item</span> <span class="nv">in</span> <span class="nv">the</span> <span class="nv">collection.</span> <span class="nv">Calls</span> <span class="nb">seq </span><span class="nv">on</span> <span class="nv">its</span>
</span><span class='line'>    <span class="nv">argument.</span> <span class="nv">If</span> <span class="nv">coll</span> <span class="nv">is</span> <span class="nv">nil</span>, <span class="nv">returns</span> <span class="nv">nil.</span>
</span></code></pre></td></tr></table></div></figure>


<h3>基本页面：src/blog/views/welcome.clj</h3>

<p>入口文件中的<code>load-views</code>函数会将指定目录下的所有文件都包含进来，这些文件中定义的是URL路由以及页面内容，以welcome.clj为例：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">blog.views.welcome</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">blog.views.common</span> <span class="ss">:as</span> <span class="nv">common</span><span class="p">]</span>
</span><span class='line'>            <span class="p">[</span><span class="nv">noir.content.getting-started</span><span class="p">])</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:use</span> <span class="p">[</span><span class="nv">noir.core</span> <span class="ss">:only</span> <span class="p">[</span><span class="nv">defpage</span><span class="p">]]</span>
</span><span class='line'>        <span class="p">[</span><span class="nv">hiccup.core</span> <span class="ss">:only</span> <span class="p">[</span><span class="nv">html</span><span class="p">]]))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="nf">defpage</span> <span class="s">&quot;/welcome&quot;</span> <span class="p">[]</span>
</span><span class='line'>         <span class="p">(</span><span class="nf">common/layout</span>
</span><span class='line'>           <span class="p">[</span><span class="ss">:p</span> <span class="s">&quot;Welcome to blog&quot;</span><span class="p">]))</span>
</span></code></pre></td></tr></table></div></figure>


<p>我们先跳过这些代码，来看看如何定义一个新的页面。将以下代码添加到welcome.clj尾部，然后执行<code>lein run</code>。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="nf">defpage</span> <span class="s">&quot;/greeting&quot;</span> <span class="p">[]</span>
</span><span class='line'>         <span class="p">(</span><span class="nf">html</span>
</span><span class='line'>           <span class="p">[</span><span class="ss">:h1</span> <span class="s">&quot;Hello, world!&quot;</span><span class="p">]))</span>
</span></code></pre></td></tr></table></div></figure>


<p>访问 http://127.0.0.1:8080/greeting 就能看到一个新的页面了，页面源码是<code>&lt;h1&gt;Hello, world!&lt;/h1&gt;</code>。</p>

<p>defpage是Noir的一个宏，用来定义URL和它返回的页面内容。URL的定义有很多其他用法，如POST方式、截取参数等，我们稍后都会用到。页面内容方面，可以直接返回字符串，如<code>(defpage "/greeting" [] "Hello, world!")</code>，也可以使用Hiccup构建HTML页面。<a href="https://github.com/weavejester/hiccup">Hiccup</a>是Noir默认的模板引擎，简单来说就是用Clojure来写HTML。一个完整的页面示例如下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">...</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:use</span> <span class="nv">...</span>
</span><span class='line'>        <span class="p">[</span><span class="nv">hiccup.page-helpers</span> <span class="ss">:only</span> <span class="p">[</span><span class="nv">html4</span><span class="p">]]))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="nf">defpage</span> <span class="s">&quot;/greeting&quot;</span> <span class="p">[]</span>
</span><span class='line'>         <span class="p">(</span><span class="nf">html4</span>
</span><span class='line'>           <span class="p">[</span><span class="ss">:head</span>
</span><span class='line'>             <span class="p">[</span><span class="ss">:title</span> <span class="s">&quot;Greeting&quot;</span><span class="p">]]</span>
</span><span class='line'>           <span class="p">[</span><span class="ss">:body</span>
</span><span class='line'>             <span class="p">[</span><span class="ss">:h1</span> <span class="s">&quot;Hello, world!&quot;</span><span class="p">]]))</span>
</span></code></pre></td></tr></table></div></figure>


<p>生成的HTML是：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
</pre></td><td class='code'><pre><code class='html'><span class='line'><span class="cp">&lt;!DOCTYPE html PUBLIC &quot;-//W3C//DTD HTML 4.01//EN&quot; &quot;http://www.w3.org/TR/html4/strict.dtd&quot;&gt;</span>
</span><span class='line'><span class="nt">&lt;html&gt;</span>
</span><span class='line'>  <span class="nt">&lt;head&gt;</span>
</span><span class='line'>    <span class="nt">&lt;title&gt;</span>Greeting<span class="nt">&lt;/title&gt;</span>
</span><span class='line'>  <span class="nt">&lt;/head&gt;</span>
</span><span class='line'>  <span class="nt">&lt;body&gt;</span>
</span><span class='line'>    <span class="nt">&lt;h1&gt;</span>Hello, world!<span class="nt">&lt;/h1&gt;</span>
</span><span class='line'>  <span class="nt">&lt;/body&gt;</span>
</span><span class='line'><span class="nt">&lt;/html&gt;</span>
</span></code></pre></td></tr></table></div></figure>


<p>其中，html和html4都是Hiccup定义的宏。html仅将接收到的参数转换为HTML代码，html4则是会添加相应版本的<code>&lt;!DOCTYPE&gt;</code>和<code>&lt;html&gt;</code>标签。要使用这些宏需要引入相应的命名空间。:only则表示只引入命名空间中特定的宏。</p>

<p>关于命名空间中出现的<code>-</code>，Clojure在编译时会自动转换成<code>_</code>，从而确保程序在JVM中运行时不会出现问题。</p>

<h3>页面模板：src/blog/views/common.clj</h3>

<p>回到系统生成的welcome.clj文件，它并没有使用html或html4，而是调用了一个<code>common/layout</code>函数。那么让我们看看common.clj中这个函数的定义：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='clojure'><span class='line'><span class="p">(</span><span class="kd">ns </span><span class="nv">blog.views.common</span>
</span><span class='line'>  <span class="p">(</span><span class="ss">:use</span> <span class="p">[</span><span class="nv">noir.core</span> <span class="ss">:only</span> <span class="p">[</span><span class="nv">defpartial</span><span class="p">]]</span>
</span><span class='line'>        <span class="p">[</span><span class="nv">hiccup.page-helpers</span> <span class="ss">:only</span> <span class="p">[</span><span class="nv">include-css</span> <span class="nv">html5</span><span class="p">]]))</span>
</span><span class='line'>
</span><span class='line'><span class="p">(</span><span class="nf">defpartial</span> <span class="nv">layout</span> <span class="p">[</span><span class="o">&amp;</span> <span class="nv">content</span><span class="p">]</span>
</span><span class='line'>            <span class="p">(</span><span class="nf">html5</span>
</span><span class='line'>              <span class="p">[</span><span class="ss">:head</span>
</span><span class='line'>               <span class="p">[</span><span class="ss">:title</span> <span class="s">&quot;blog&quot;</span><span class="p">]</span>
</span><span class='line'>               <span class="p">(</span><span class="nf">include-css</span> <span class="s">&quot;/css/reset.css&quot;</span><span class="p">)]</span>
</span><span class='line'>              <span class="p">[</span><span class="ss">:body</span>
</span><span class='line'>               <span class="p">[</span><span class="ss">:div#wrapper</span>
</span><span class='line'>                <span class="nv">content</span><span class="p">]]))</span>
</span></code></pre></td></tr></table></div></figure>


<p>defpartial是Noir的一个宏，用来定义一段可复用的HTML代码。当然我们也可以将其定义为一个函数（用<code>defn</code>替换掉<code>defpartial</code>），不会有什么区别。官方文档的解释是使用defpartial会比较容易辨认。</p>

<p><code>include-css</code>是一个函数，用来生成<code>&lt;link&gt;</code>标签。<code>[:div#wrapper ...]</code>会生成<code>&lt;div id="wrapper"&gt;...&lt;/div&gt;</code>。更多Hiccup的语法可以到<a href="http://www.webnoir.org/tutorials/html">这个页面</a>浏览一下。</p>

<h3>默认首页：noir.content.getting-started</h3>

<p>我们在代码中并没有看到<code>(defpage "/" [] ...)</code>这样的定义，那为什么网站根目录会出现一个默认页面呢？答案在<code>noir.content.getting-started</code>这个命名空间中，可以<a href="https://github.com/noir-clojure/noir/blob/master/src/noir/content/getting_started.clj">点击这里</a>查看它的源码。要取消这个默认页面，可以在welcome.clj的:require中将其删除。</p>

<h3>静态资源：src/resources/public</h3>

<p>Noir默认对src/resources/public目录下的文件做了路由，因此当有一个资源文件位于src/resources/public/css/reset.css时，可以通过http://127.0.0.1:8080/css/reset.css访问。</p>

<p>值得一提的是，Noir项目本身依赖于两个开源项目：<a href="https://github.com/ring-clojure/ring">ring</a>和<a href="https://github.com/weavejester/compojure/tree/master/src">compojure</a>，前者对HTTP请求进行了封装，提供了一套类似Python WSGI的API；后者则是专门提供URL路由功能的类库。如对静态资源的路由，实质上Noir是调用了compojure提供的resources函数，函数中又调用ring提供的GET、wrap-file-info等函数响应请求。</p>

<h2>小结</h2>

<p>本章讲述了Clojure环境的搭建，特别是项目管理工具<code>lein</code>的一般使用。通过对Noir项目骨架的分析，我们一窥Clojure的语法，接触了变量、直接量、函数、宏、命名空间的一些用法，并能结合Noir和Hiccup写出简单的页面来。下一章我们将讲解如何使用Noir编写表单页面进行交互，以及Clojure如何连接数据库，对博文进行增删改查等操作。</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[开源软件架构 - 卷1：第14章 Python打包工具]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/11/18/aosa-python-packaging/"/>
    <updated>2012-11-18T19:20:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/11/18/aosa-python-packaging</id>
    <content type="html"><![CDATA[<p>作者：<a href="http://www.aosabook.org/en/intro1.html#ziade-tarek">Tarek Ziadé</a>，翻译：<a href="mailto:zhangji87@gmail.com">张吉</a></p>

<p>原文：<a href="http://www.aosabook.org/en/packaging.html">http://www.aosabook.org/en/packaging.html</a></p>

<h2>14.1 简介</h2>

<p>对于如何安装软件，目前有两种思想流派。第一种是说软件应该自给自足，不依赖于其它任何部件，这点在Windows和Mac OS X系统中很流行。这种方式简化了软件的管理：每个软件都有自己独立的“领域”，安装和卸载它们不会对操作系统产生影响。如果软件依赖一项不常见的类库，那么这个类库一定是包含在软件安装包之中的。</p>

<p>第二种流派，主要在类Linux的操作系统中盛行，即软件应该是由一个个独立的、小型的软件包组成的。类库被包含在软件包中，包与包之间可以有依赖关系。安装软件时需要查找和安装它所依赖的其他特定版本的软件包。这些依赖包通常是从一个包含所有软件包的中央仓库中获取的。这种理念也催生了Linux发行版中那些复杂的依赖管理工具，如<code>dpkg</code>和<code>RPM</code>。它们会跟踪软件包的依赖关系，并防止两个软件使用了版本相冲突的第三方包。</p>

<p>以上两种流派各有优劣。高度模块化的系统可以使得更新和替换某个软件包变的非常方便，因为每个类库都只有一份，所有依赖于它的应用程序都能因此受益。比如，修复某个类库的安全漏洞可以立刻应用到所有程序中，而如果应用程序使用了自带的类库，那安全更新就很难应用进去了，特别是在类库版本不一致的情况下更难处理。</p>

<p>不过这种“模块化”也被一些开发者视为缺点，因为他们无法控制应用程序的依赖关系。他们希望提供一个独立和稳定的软件运行环境，这样就不会在系统升级后遭遇各种依赖方面的问题。</p>

<!-- more -->


<p>在安装程序中包含所有依赖包还有一个优点：便于跨平台。有些项目在这点上做到了极致，它们将所有和操作系统的交互都封装了起来，在一个独立的目录中运行，甚至包括日志文件的记录位置。</p>

<p>Python的打包系统使用的是第二种设计思想，并尽可能地方便开发者、管理员、用户对软件的管理。不幸的是，这种方式导致了种种问题：错综复杂的版本结构、混乱的数据文件、难以重新打包等等。三年前，我和其他一些Python开发者决定研究解决这个问题，我们自称为“打包别动队”，本文就是讲述我们在这个问题上做出的努力和取得的成果。</p>

<h3>术语</h3>

<p>在Python中， <em>包</em> 表示一个包含Python文件的目录。Python文件被称为 <em>模块</em> ，这样一来，使用“包”这个单词就显得有些模糊了，因为它常常用来表示某个项目的 <em>发行版本</em> 。</p>

<p>Python开发者有时也对此表示不能理解。为了更清晰地进行表述，我们用“Python包（package）”来表示一个包含Python文件的目录，用“发行版本（release）”来表示某个项目的特定版本，用“发布包（distribution）”来表示某个发行版本的源码或二进制文件，通常是Tar包或Zip文件的形式。</p>

<h2>14.2 Python开发者的困境</h2>

<p>大多数Python开发者希望自己的程序能够在任何环境中运行。他们还希望自己的软件既能使用标准的Python类库，又能使用依赖于特定系统类型的类库。但除非开发者使用现有的各种打包工具生成不同的软件包，否则他们打出的软件安装包就必须在一个安装有Python环境的系统中运行。这样的软件包还希望做到以下几点：</p>

<ul>
<li>其他人可以针对不同的目标系统对这个软件重新打包；</li>
<li>软件所依赖的包也能够针对不同的目标系统进行重新打包；</li>
<li>系统依赖项能够被清晰地描述出来。</li>
</ul>


<p>要做到以上几点往往是不可能的。举例来说，Plone这一功能全面的CMS系统，使用了上百个纯Python语言编写的类库，而这些类库并不一定在所有的打包系统中提供。这就意味着Plone必须将它所依赖的软件包都集成到自己的安装包中。要做到这一点，他们选择使用<code>zc.buildout</code>这一工具，它能够将所有的依赖包都收集起来，生成一个完整的应用程序文件，在独立的目录中运行。它事实上是一个二进制的软件包，因为所有C语言代码都已经编译好了。</p>

<p>这对开发者来说是福音：他们只需要描述好依赖关系，然后借助<code>zc.buildout</code>来发布自己的程序即可。但正如上文所言，这种发布方式在系统层面构筑了一层屏障，这让大多数Linux系统管理员非常恼火。Windows管理员不会在乎这些，但CentOS和Debian管理员则会，因为按照他们的管理原则，系统中的所有文件都应该被注册和归类到现有的管理工具中。</p>

<p>这些管理员会想要将你的软件按照他们自己的标准重新打包。问题在于：Python有没有这样的打包工具，能够自动地按照新的标准重新打包？如果有，那么Python的任何软件和类库就能够针对不同的目标系统进行打包，而不需要额外的工作。这里，“自动”一词并不是说打包过程可以完全由脚本来完成——这点上<code>RPM</code>和<code>dpkg</code>的使用者已经证实是不可能的了，因为他们总会需要增加额外的信息来重新打包。他们还会告诉你，在重新打包的过程中会遇到一些开发者没有遵守基本打包原则的情况。</p>

<p>我们来举一个实际例子，如何通过使用现有的Python打包工具来惹恼那些想要重新打包的管理员：在发布一个名为“MathUtils”的软件包时使用“Fumanchu”这样的版本号名字。撰写这个类库的数学家想用自家猫咪的名字来作为版本号，但是管理员怎么可能知道“Fumanchu”是他家第二只猫的名字，第一只猫叫做“Phil”，所以“Fumanchu”版本要比“Phil”版本来得高？</p>

<p>可能这个例子有些极端，但是在现有的打包工具和规范中是可能发生的。最坏的情况是<code>easy_install</code>和<code>pip</code>使用自己的一套标准来追踪已安装的文件，并使用字母顺序来比较“Fumanchu”和“Phil”的版本高低。</p>

<p>另一个问题是如何处理数据文件。比如，如果你的软件使用了SQLite数据库，安装时被放置在包目录中，那么在程序运行时，系统会阻止你对其进行读写操作。这样做还会破坏Linux系统的一项惯例，即<code>/var</code>目录下的数据文件是需要进行备份的。</p>

<p>在现实环境中，系统管理员需要能够将你的文件放置到他们想要的地方，并且不破坏程序的完整性，这就需要你来告诉他们各类文件都是做什么用的。让我们换一种方式来表述刚才的问题：Python是否有这样一种打包工具，它可以提供各类信息，足以让第三方打包工具能据此重新进行打包，而不需要阅读软件的源码？</p>

<h2>14.3 现有的打包管理架构</h2>

<p>Python标准库中提供的<code>Distutils</code>打包工具充斥了上述的种种问题，但由于它是一种标准，所以人们要么继续忍受并使用它，或者转向更先进的工具<code>Setuptools</code>，它在Distutils之上提供了一些高级特性。另外还有<code>Distribute</code>，它是<code>Setuptools</code>的衍生版本。<code>Pip</code>则是一种更为高级的安装工具，它依赖于<code>Setuptools</code>。</p>

<p>但是，这些工具都源自于<code>Distutils</code>，并继承了它的种种问题。有人也想过要改进<code>Distutils</code>本身，但是由于它的使用范围已经很广很广，任何小的改动都会对Python软件包的整个生态系统造成冲击。</p>

<p>所以，我们决定冻结<code>Distutils</code>的代码，并开始研发<code>Distutils2</code>，不去考虑向前兼容的问题。为了解释我们所做的改动，首先让我们近距离观察一下<code>Distutils</code>。</p>

<h3>14.3.1 Distutils基础及设计缺陷</h3>

<p><code>Distutils</code>由一些命令组成，每条命令都是一个包含了<code>run</code>方法的类，可以附加若干参数进行调用。<code>Distutils</code>还提供了一个名为<code>Distribution</code>的类，它包含了一些全局变量，可供其他命令使用。</p>

<p>当要使用<code>Distutils</code>时，Python开发者需要在项目中添加一个模块，通常命名为<code>setup.py</code>。这个模块会调用<code>Distutils</code>的入口函数：<code>setup</code>。这个函数有很多参数，这些参数会被<code>Distribution</code>实例保存起来，供后续使用。下面这个例子中我们指定了一些常用的参数，如项目名称和版本，它所包含的模块等：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">from</span> <span class="nn">distutils.core</span> <span class="kn">import</span> <span class="n">setup</span>
</span><span class='line'>
</span><span class='line'><span class="n">setup</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">&#39;MyProject&#39;</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="s">&#39;1.0&#39;</span><span class="p">,</span> <span class="n">py_modules</span><span class="o">=</span><span class="p">[</span><span class="s">&#39;mycode.py&#39;</span><span class="p">])</span>
</span></code></pre></td></tr></table></div></figure>


<p>这个模块可以用来执行<code>Distutils</code>的各种命令，如<code>sdist</code>。这条命令会在<code>dist</code>目录中创建一个源代码发布包：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>python setup.py sdist
</span></code></pre></td></tr></table></div></figure>


<p>这个模块还可以执行<code>install</code>命令：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>python setup.py install
</span></code></pre></td></tr></table></div></figure>


<p><code>Distutils</code>还提供了一些其他命令：</p>

<ul>
<li><code>upload</code> 将发布包上传至在线仓库</li>
<li><code>register</code> 向在线仓库注册项目的基本信息，而不上传发布包</li>
<li><code>bdist</code> 创建二进制发布包</li>
<li><code>bdist_msi</code> 创建<code>.msi</code>安装包，供Windows系统使用</li>
</ul>


<p>我们还可以使用其他一些命令来获取项目的基本信息。</p>

<p>所以在安装或获取应用程序信息时都是通过这个文件调用<code>Distutils</code>实现的，如获取项目名称：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>python setup.py --name
</span><span class='line'>MyProject
</span></code></pre></td></tr></table></div></figure>


<p><code>setup.py</code>是一个项目的入口，可以通过它对项目进行构建、打包、发布、安装等操作。开发者通过这个函数的参数信息来描述自己的项目，并使用它进行各种打包任务。这个文件同样用于在目标系统中安装软件。</p>

<p><img src="http://www.aosabook.org/images/packaging/setup-py.png" alt="图14.1 安装" /></p>

<p>图14.1 安装</p>

<p>然而，使用同一个文件来对项目进行打包、发布、以及安装，是<code>Distutils</code>的主要缺点。例如，你需要查看<code>lxml</code>项目的名称属性，<code>setup.py</code>会执行很多其他无关的操作，而不是简单返回一个字符串：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>python setup.py --name
</span><span class='line'>Building lxml version 2.2.
</span><span class='line'>NOTE: Trying to build without Cython, pre-generated <span class="s1">&#39;src/lxml/lxml.etree.c&#39;</span>
</span><span class='line'>needs to be available.
</span><span class='line'>Using build configuration of libxslt 1.1.26
</span><span class='line'>Building against libxml2/libxslt in the following directory: /usr/lib/lxml
</span></code></pre></td></tr></table></div></figure>


<p>在有些项目中它甚至会执行失败，因为开发者默认为<code>setup.py</code>只是用来安装软件的，而其他一些<code>Distutils</code>功能只在开发过程中使用。因此，<code>setup.py</code>的角色太多，容易引起他人的困惑。</p>

<h3>14.3.2 元信息和PyPI</h3>

<p><code>Distutils</code>在构建发布包时会创建一个<code>Metadata</code>文件。这个文件是按照PEP314<sup>1</sup>编写的，包含了一些常见的项目信息，包括名称、版本等，主要有以下几项：</p>

<ul>
<li><code>Name</code>：项目名称</li>
<li><code>Version</code>：发布版本号</li>
<li><code>Summary</code>：项目简介</li>
<li><code>Description</code>：项目详情</li>
<li><code>Home-Page</code>：项目主页</li>
<li><code>Author</code>：作者</li>
<li><code>Classifers</code>：项目类别。Python为不同的发行协议、发布版本（beta，alpha，final）等提供了不同的类别。</li>
<li><code>Requires</code>，<code>Provides</code>，<code>Obsoletes</code>：描述项目依赖信息</li>
</ul>


<p>这些信息一般都能移植到其他打包系统中。</p>

<p>Python项目索引（Python Package Index，简称PyPI<sup>2</sup>），是一个类似CPAN的中央软件包仓库，可以调用<code>Distutils</code>的<code>register</code>和<code>upload</code>命令来注册和发布项目。<code>register</code>命令会构建<code>Metadata</code>文件并传送给PyPI，让访问者和安装工具能够浏览和搜索。</p>

<p><img src="http://www.aosabook.org/images/packaging/pypi.png" alt="图14.2：PyPI仓库" /></p>

<p>图14.2：PyPI仓库</p>

<p>你可以通过<code>Classifies</code>（类别）来浏览，获取项目作者的名字和主页。同时，<code>Requires</code>可以用来定义Python模块的依赖关系。<code>requires</code>选项可以向元信息文件的<code>Requires</code>字段添加信息：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">from</span> <span class="nn">distutils.core</span> <span class="kn">import</span> <span class="n">setup</span>
</span><span class='line'>
</span><span class='line'><span class="n">setup</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">&#39;foo&#39;</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="s">&#39;1.0&#39;</span><span class="p">,</span> <span class="n">requires</span><span class="o">=</span><span class="p">[</span><span class="s">&#39;ldap&#39;</span><span class="p">])</span>
</span></code></pre></td></tr></table></div></figure>


<p>这里声明了对<code>ldap</code>模块的依赖，这种依赖并没有实际效力，因为没有安装工具会保证这个模块真实存在。如果说Python代码中会使用类似Perl的<code>require</code>关键字来定义依赖关系，那还有些作用，因为这时安装工具会检索PyPI上的信息并进行安装，其实这也就是CPAN的做法。但是对于Python来说，<code>ldap</code>模块可以存在于任何项目之中，因为<code>Distutils</code>是允许开发者发布一个包含多个模块的软件的，所以这里的元信息字段并无太大作用。</p>

<p><code>Metadata</code>的另一个缺点是，因为它是由Python脚本创建的，所以会根据脚本执行环境的不同而产生特定信息。比如，运行在Windows环境下的一个项目会在<code>setup.py</code>文件中有以下描述：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">from</span> <span class="nn">distutils.core</span> <span class="kn">import</span> <span class="n">setup</span>
</span><span class='line'>
</span><span class='line'><span class="n">setup</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">&#39;foo&#39;</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="s">&#39;1.0&#39;</span><span class="p">,</span> <span class="n">requires</span><span class="o">=</span><span class="p">[</span><span class="s">&#39;win32com&#39;</span><span class="p">])</span>
</span></code></pre></td></tr></table></div></figure>


<p>这样配置相当于是默认该项目只会运行在Windows环境下，即使它可能提供了跨平台的方案。一种解决方法是根据不同的平台来指定<code>requires</code>参数：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">from</span> <span class="nn">distutils.core</span> <span class="kn">import</span> <span class="n">setup</span>
</span><span class='line'><span class="kn">import</span> <span class="nn">sys</span>
</span><span class='line'>
</span><span class='line'><span class="k">if</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span> <span class="o">==</span> <span class="s">&#39;win32&#39;</span><span class="p">:</span>
</span><span class='line'>    <span class="n">setup</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">&#39;foo&#39;</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="s">&#39;1.0&#39;</span><span class="p">,</span> <span class="n">requires</span><span class="o">=</span><span class="p">[</span><span class="s">&#39;win32com&#39;</span><span class="p">])</span>
</span><span class='line'><span class="k">else</span><span class="p">:</span>
</span><span class='line'>    <span class="n">setup</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">&#39;foo&#39;</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="s">&#39;1.0&#39;</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>但这种做法往往会让事情更糟。要注意，这个脚本是用来将项目的源码包发布到PyPI上的，这样写就说明它向PyPI上传的<code>Metadata</code>文件会因为该脚本运行环境的不同而不同。换句话说，这使得我们无法在元信息文件中看出这个项目依赖于特定的平台。</p>

<h3>14.3.3 PyPI的架构设计</h3>

<p><img src="http://www.aosabook.org/images/packaging/pypi-workflow.png" alt="图14.3 PyPI工作流" /></p>

<p>图14.3 PyPI工作流</p>

<p>如上文所述，PyPI是一个Python项目的中央仓库，人们可以通过不同的类别来搜索已有的项目，也可以创建自己的项目。人们可以上传项目源码和二进制文件，供其他人下载使用或研究。同时，PyPI还提供了相应的Web服务，让安装工具可以调用它来检索和下载文件。</p>

<h4>注册项目并上传发布包</h4>

<p>我们可以使用<code>Distutils</code>的<code>register</code>命令在PyPI中注册一个项目。这个命令会根据项目的元信息生成一个POST请求。该请求会包含验证信息，PyPI使用HTTP基本验证来确保所有的项目都和一个注册用户相关联。验证信息保存在<code>Distutils</code>的配置文件中，或在每次执行<code>register</code>命令时提示用户输入。以下是一个使用示例：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>python setup.py register
</span><span class='line'>running register
</span><span class='line'>Registering MPTools to http://pypi.python.org/pypi
</span><span class='line'>Server response <span class="o">(</span>200<span class="o">)</span>: OK
</span></code></pre></td></tr></table></div></figure>


<p>每个注册项目都会产生一个HTML页面，上面包含了它的元信息。开发者可以使用<code>upload</code>命令将发布包上传至PyPI：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>python setup.py sdist upload
</span><span class='line'>running sdist
</span><span class='line'>…
</span><span class='line'>running upload
</span><span class='line'>Submitting dist/mopytools-0.1.tar.gz to http://pypi.python.org/pypi
</span><span class='line'>Server response <span class="o">(</span>200<span class="o">)</span>: OK
</span></code></pre></td></tr></table></div></figure>


<p>如果开发者不想将代码上传至PyPI，可以使用元信息中的<code>Download-URL</code>属性来指定一个外部链接，供用户下载。</p>

<h4>检索PyPI</h4>

<p>除了在页面中检索项目，PyPI还提供了两个接口供程序调用：简单索引协议和XML-PRC API。</p>

<p>简单索引协议的地址是<code>http://pypi.python.org/simple/</code>，它包含了一个链接列表，指向所有的注册项目：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class='html'><span class='line'><span class="nt">&lt;html&gt;&lt;head&gt;&lt;title&gt;</span>Simple Index<span class="nt">&lt;/title&gt;&lt;/head&gt;&lt;body&gt;</span>
</span><span class='line'>⋮    ⋮    ⋮
</span><span class='line'><span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">&#39;MontyLingua/&#39;</span><span class="nt">&gt;</span>MontyLingua<span class="nt">&lt;/a&gt;&lt;br/&gt;</span>
</span><span class='line'><span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">&#39;mootiro_web/&#39;</span><span class="nt">&gt;</span>mootiro_web<span class="nt">&lt;/a&gt;&lt;br/&gt;</span>
</span><span class='line'><span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">&#39;Mopidy/&#39;</span><span class="nt">&gt;</span>Mopidy<span class="nt">&lt;/a&gt;&lt;br/&gt;</span>
</span><span class='line'><span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">&#39;mopowg/&#39;</span><span class="nt">&gt;</span>mopowg<span class="nt">&lt;/a&gt;&lt;br/&gt;</span>
</span><span class='line'><span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">&#39;MOPPY/&#39;</span><span class="nt">&gt;</span>MOPPY<span class="nt">&lt;/a&gt;&lt;br/&gt;</span>
</span><span class='line'><span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">&#39;MPTools/&#39;</span><span class="nt">&gt;</span>MPTools<span class="nt">&lt;/a&gt;&lt;br/&gt;</span>
</span><span class='line'><span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">&#39;morbid/&#39;</span><span class="nt">&gt;</span>morbid<span class="nt">&lt;/a&gt;&lt;br/&gt;</span>
</span><span class='line'><span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">&#39;Morelia/&#39;</span><span class="nt">&gt;</span>Morelia<span class="nt">&lt;/a&gt;&lt;br/&gt;</span>
</span><span class='line'><span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">&#39;morse/&#39;</span><span class="nt">&gt;</span>morse<span class="nt">&lt;/a&gt;&lt;br/&gt;</span>
</span><span class='line'>⋮    ⋮    ⋮
</span><span class='line'><span class="nt">&lt;/body&gt;&lt;/html&gt;</span>
</span></code></pre></td></tr></table></div></figure>


<p>如MPTools项目对应的<code>MPTools/</code>目录，它所指向的路径会包含以下内容：</p>

<ul>
<li>所有发布包的地址</li>
<li>在<code>Metadata</code>中定义的项目网站地址，包含所有版本</li>
<li>下载地址（<code>Download-URL</code>），同样包含所有版本</li>
</ul>


<p>以MPTools项目为例：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='html'><span class='line'><span class="nt">&lt;html&gt;&lt;head&gt;&lt;title&gt;</span>Links for MPTools<span class="nt">&lt;/title&gt;&lt;/head&gt;</span>
</span><span class='line'><span class="nt">&lt;body&gt;&lt;h1&gt;</span>Links for MPTools<span class="nt">&lt;/h1&gt;</span>
</span><span class='line'><span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">&quot;../../packages/source/M/MPTools/MPTools-0.1.tar.gz&quot;</span><span class="nt">&gt;</span>MPTools-0.1.tar.gz<span class="nt">&lt;/a&gt;&lt;br/&gt;</span>
</span><span class='line'><span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">&quot;http://bitbucket.org/tarek/mopytools&quot;</span> <span class="na">rel=</span><span class="s">&quot;homepage&quot;</span><span class="nt">&gt;</span>0.1 home_page<span class="nt">&lt;/a&gt;&lt;br/&gt;</span>
</span><span class='line'><span class="nt">&lt;/body&gt;&lt;/html&gt;</span>
</span></code></pre></td></tr></table></div></figure>


<p>安装工具可以通过访问这个索引来查找项目的发布包，或者检查<code>http://pypi.python.org/simple/PROJECT_NAME/</code>是否存在。</p>

<p>但是，这个协议主要有两个缺陷。首先，PyPI目前还是单台服务器。虽然很多用户会自己搭建镜像，但过去两年中曾发生过几次PyPI无法访问的情况，用户无法下载依赖包，导致项目构建出现问题。比如说，在构建一个Plone项目时，需要向PyPI发送近百次请求。所以PyPI在这里成为了单点故障。</p>

<p>其次，当项目的发布包没有保存在PyPI中，而是通过<code>Download-URL</code>指向了其他地址，安装工具就需要重定向到这个地址下载发布包。这种情况也会增加安装过程的不稳定性。</p>

<p>简单索引协议只是提供给安装工具一个项目列表，并不包含项目元信息。可以通过PyPI的XML-RPC API来获取项目元信息：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">xmlrpclib</span>
</span><span class='line'><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">pprint</span>
</span><span class='line'><span class="o">&gt;&gt;&gt;</span> <span class="n">client</span> <span class="o">=</span> <span class="n">xmlrpclib</span><span class="o">.</span><span class="n">ServerProxy</span><span class="p">(</span><span class="s">&#39;http://pypi.python.org/pypi&#39;</span><span class="p">)</span>
</span><span class='line'><span class="o">&gt;&gt;&gt;</span> <span class="n">client</span><span class="o">.</span><span class="n">package_releases</span><span class="p">(</span><span class="s">&#39;MPTools&#39;</span><span class="p">)</span>
</span><span class='line'><span class="p">[</span><span class="s">&#39;0.1&#39;</span><span class="p">]</span>
</span><span class='line'><span class="o">&gt;&gt;&gt;</span> <span class="n">pprint</span><span class="o">.</span><span class="n">pprint</span><span class="p">(</span><span class="n">client</span><span class="o">.</span><span class="n">release_urls</span><span class="p">(</span><span class="s">&#39;MPTools&#39;</span><span class="p">,</span> <span class="s">&#39;0.1&#39;</span><span class="p">))</span>
</span><span class='line'><span class="p">[{</span><span class="s">&#39;comment_text&#39;</span><span class="p">:</span> <span class="o">&amp;</span><span class="n">rquot</span><span class="p">;,</span>
</span><span class='line'><span class="s">&#39;downloads&#39;</span><span class="p">:</span> <span class="mi">28</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;filename&#39;</span><span class="p">:</span> <span class="s">&#39;MPTools-0.1.tar.gz&#39;</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;has_sig&#39;</span><span class="p">:</span> <span class="bp">False</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;md5_digest&#39;</span><span class="p">:</span> <span class="s">&#39;6b06752d62c4bffe1fb65cd5c9b7111a&#39;</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;packagetype&#39;</span><span class="p">:</span> <span class="s">&#39;sdist&#39;</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;python_version&#39;</span><span class="p">:</span> <span class="s">&#39;source&#39;</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;size&#39;</span><span class="p">:</span> <span class="mi">3684</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;upload_time&#39;</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">DateTime</span> <span class="s">&#39;20110204T09:37:12&#39;</span> <span class="n">at</span> <span class="n">f4da28</span><span class="o">&gt;</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;url&#39;</span><span class="p">:</span> <span class="s">&#39;http://pypi.python.org/packages/source/M/MPTools/MPTools-0.1.tar.gz&#39;</span><span class="p">}]</span>
</span><span class='line'><span class="o">&gt;&gt;&gt;</span> <span class="n">pprint</span><span class="o">.</span><span class="n">pprint</span><span class="p">(</span><span class="n">client</span><span class="o">.</span><span class="n">release_data</span><span class="p">(</span><span class="s">&#39;MPTools&#39;</span><span class="p">,</span> <span class="s">&#39;0.1&#39;</span><span class="p">))</span>
</span><span class='line'><span class="p">{</span><span class="s">&#39;author&#39;</span><span class="p">:</span> <span class="s">&#39;Tarek Ziade&#39;</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;author_email&#39;</span><span class="p">:</span> <span class="s">&#39;tarek@mozilla.com&#39;</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;classifiers&#39;</span><span class="p">:</span> <span class="p">[],</span>
</span><span class='line'><span class="s">&#39;description&#39;</span><span class="p">:</span> <span class="s">&#39;UNKNOWN&#39;</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;download_url&#39;</span><span class="p">:</span> <span class="s">&#39;UNKNOWN&#39;</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;home_page&#39;</span><span class="p">:</span> <span class="s">&#39;http://bitbucket.org/tarek/mopytools&#39;</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;keywords&#39;</span><span class="p">:</span> <span class="bp">None</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;license&#39;</span><span class="p">:</span> <span class="s">&#39;UNKNOWN&#39;</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;maintainer&#39;</span><span class="p">:</span> <span class="bp">None</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;maintainer_email&#39;</span><span class="p">:</span> <span class="bp">None</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;name&#39;</span><span class="p">:</span> <span class="s">&#39;MPTools&#39;</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;package_url&#39;</span><span class="p">:</span> <span class="s">&#39;http://pypi.python.org/pypi/MPTools&#39;</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;platform&#39;</span><span class="p">:</span> <span class="s">&#39;UNKNOWN&#39;</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;release_url&#39;</span><span class="p">:</span> <span class="s">&#39;http://pypi.python.org/pypi/MPTools/0.1&#39;</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;requires_python&#39;</span><span class="p">:</span> <span class="bp">None</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;stable_version&#39;</span><span class="p">:</span> <span class="bp">None</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;summary&#39;</span><span class="p">:</span> <span class="s">&#39;Set of tools to build Mozilla Services apps&#39;</span><span class="p">,</span>
</span><span class='line'><span class="s">&#39;version&#39;</span><span class="p">:</span> <span class="s">&#39;0.1&#39;</span><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>这种方式的问题在于，项目元信息原本就能以静态文件的方式在简单索引协议中提供，这样可以简化安装工具的复杂性，也可以减少PyPI服务的请求数。对于诸如下载数量这样的动态数据，可以在其他接口中提供。用两种服务来获取所有的静态内容，显然不太合理。</p>

<h3>14.3.4 Python安装目录的结构</h3>

<p>在使用<code>python setup.py install</code>安装一个Python项目后，<code>Distutils</code>这一Python核心类库会负责将程序代码复制到目标系统的相应位置。</p>

<ul>
<li><em>Python包</em> 和模块会被安装到Python解释器程序所在的目录中，并随解释器启动：Ubuntu系统中会安装到<code>/usr/local/lib/python2.6/dist-packages/</code>，Fedora则是<code>/usr/local/lib/python2.6/sites-packages/</code>。</li>
<li>项目中的 <em>数据文件</em> 可以被安装到任何位置。</li>
<li><em>可执行文件</em> 会被安装到系统的<code>bin</code>目录下，依平台类型而定，可能是<code>/usr/local/bin</code>，或是其它指定的目录。</li>
</ul>


<p>从Python2.5开始，项目的元信息文件会随模块和包一起发布，名称为<code>project-version.egg-info</code>。比如，<code>virtualenv</code>项目会有一个<code>virtualenv-1.4.9.egg-info</code>文件。这些元信息文件可以被视为一个已安装项目的数据库，因为可以通过遍历其中的内容来获取已安装的项目和版本。但是，<code>Distutils</code>并没有记录项目所安装的文件列表，也就是说，我们无法彻底删除安装某个项目后产生的所有文件。可惜的是，<code>install</code>命令本身是提供了一个名为<code>--record</code>的参数的，可以将已安装的文件列表记录在文本文件中，但是这个参数并没有默认开启，而且<code>Distutils</code>的文档中几乎没有提及这个参数。</p>

<h3>14.3.5 Setuptools、Pip等工具</h3>

<p>正如介绍中所提到的，有些项目已经在尝试修复<code>Distutils</code>的问题，并取得了一些成功。</p>

<h4>依赖问题</h4>

<p>PyPI允许开发者在发布的项目中包含多个模块，还允许项目通过定义<code>Require</code>属性来声明模块级别的依赖。这两种做法都是合理的，但是同时使用就会很糟糕。</p>

<p>正确的做法应该是定义项目级别的依赖，这也是<code>Setuptools</code>在<code>Distutils</code>之上附加的一个特性。它还提供了一个名为<code>easy_install</code>的脚本来从PyPI上自动获取和安装依赖项。在实际生产中，模块级别的依赖并没有真正被使用，更多人倾向于使用<code>Setuptools</code>。然而，这些特性只是针对<code>Setuptools</code>的，并没有被<code>Distutils</code>或PyPI所接受，所以<code>Setuptools</code>实质上是一个构建在错误设计上的仿冒品。</p>

<p><code>easy_install</code>需要下载项目的压缩文档，执行<code>setup.py</code>来获取元信息，并对每个依赖项进行相同的操作。项目的依赖树会随着软件包的下载逐步勾画出来。</p>

<p>虽然PyPI上可以直接浏览项目元信息，但是<code>easy_install</code>还是需要下载所有的软件包，因为上文提到过，PyPI上的项目元信息很可能和上传时所使用的平台有关，从而和目标系统有所差异。但是这种一次性安装项目依赖的做法已经能够解决90%的问题了，的确是个很不错的特性。这也是为什么<code>Setuptools</code>被广泛采用的原因。然而，它还是有以下一些问题：</p>

<ul>
<li>如果某一个依赖项安装失败，它并没有提供回滚的选项，因此系统会处于一个不可用的状态。</li>
<li>项目依赖树是在安装一个个软件包时构建出来的，因此当其中两个依赖项产生冲突时，系统也会变的不可用。</li>
</ul>


<h4>卸载的问题</h4>

<p>虽然<code>Setuptools</code>可以在元信息中记录已安装的文件，但它并没有提供卸载功能。另一个工具<code>Pip</code>，它通过扩展<code>Setuptools</code>的元信息来记录已安装的文件，从而能够进行卸载操作。但是，这组信息又是一种自定义的内容，因此一个Python项目很可能包含四种不同的元信息：</p>

<ul>
<li><code>Distutils</code>的<code>egg-info</code>，一个单一的文件；</li>
<li><code>Setuptools</code>的<code>egg-info</code>，一个目录，记录了<code>Setuptools</code>特定的元信息；</li>
<li><code>Pip</code>的<code>egg-info</code>，是后者的扩展；</li>
<li>其它由打包系统产生的信息。</li>
</ul>


<h3>14.3.6 数据文件如何处理？</h3>

<p>在<code>Distutils</code>中，数据文件可以被安装在任意位置。你可以像这样在<code>setup.py</code>中定义一个项目的数据文件：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">setup</span><span class="p">(</span><span class="err">…</span><span class="p">,</span>
</span><span class='line'>  <span class="n">packages</span><span class="o">=</span><span class="p">[</span><span class="s">&#39;mypkg&#39;</span><span class="p">],</span>
</span><span class='line'>  <span class="n">package_dir</span><span class="o">=</span><span class="p">{</span><span class="s">&#39;mypkg&#39;</span><span class="p">:</span> <span class="s">&#39;src/mypkg&#39;</span><span class="p">},</span>
</span><span class='line'>  <span class="n">package_data</span><span class="o">=</span><span class="p">{</span><span class="s">&#39;mypkg&#39;</span><span class="p">:</span> <span class="p">[</span><span class="s">&#39;data/*.dat&#39;</span><span class="p">]},</span>
</span><span class='line'>  <span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>那么，<code>mypkg</code>项目中所有以<code>.dat</code>为扩展名的文件都会被包含在发布包中，并随Python代码安装到目标系统。</p>

<p>对于需要安装到项目目录之外的数据文件，可以进行如下配置。他们随项目一起打包，并安装到指定的目录中：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">setup</span><span class="p">(</span><span class="err">…</span><span class="p">,</span>
</span><span class='line'>    <span class="n">data_files</span><span class="o">=</span><span class="p">[(</span><span class="s">&#39;bitmaps&#39;</span><span class="p">,</span> <span class="p">[</span><span class="s">&#39;bm/b1.gif&#39;</span><span class="p">,</span> <span class="s">&#39;bm/b2.gif&#39;</span><span class="p">]),</span>
</span><span class='line'>                <span class="p">(</span><span class="s">&#39;config&#39;</span><span class="p">,</span> <span class="p">[</span><span class="s">&#39;cfg/data.cfg&#39;</span><span class="p">]),</span>
</span><span class='line'>                <span class="p">(</span><span class="s">&#39;/etc/init.d&#39;</span><span class="p">,</span> <span class="p">[</span><span class="s">&#39;init-script&#39;</span><span class="p">])]</span>
</span><span class='line'>    <span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>这对系统打包人员来说简直是噩梦：</p>

<ul>
<li>元信息中并不包含数据文件的信息，因此打包人员需要阅读<code>setup.py</code>文件，甚至是研究项目源码来获取这些信息。</li>
<li>不应该由开发人员来决定项目数据文件应该安装到目标系统的哪个位置。</li>
<li>数据文件没有区分类型，图片、帮助文件等都被视为同等来处理。</li>
</ul>


<p>打包人员在对项目进行打包时只能去根据目标系统的情况来修改<code>setup.py</code>文件，从而让软件包能够顺利安装。要做到这一点，他就需要阅读程序代码，修改所有用到这些文件的地方。<code>Setuptools</code>和<code>Pip</code>并没有解决这一问题。</p>

<h2>14.4 改进标准</h2>

<p>所以最后我们得到的是这样一个打包系统：所有功能都由一个模块提供，项目的元信息不完整，无法描述清楚项目中包含的所有内容。现在就让我们来做些改进。</p>

<h3>14.4.1 元信息</h3>

<p>首先，我们要修正<code>Metadata</code>标准中的内容。PEP 345定义了一个新的标准，它包含以下内容：</p>

<ul>
<li>更合理的版本定义方式</li>
<li>项目级别的依赖关系</li>
<li>使用一种静态的方式描述平台相关的属性</li>
</ul>


<h4>版本</h4>

<p>元信息标准的目标之一是能够让Python包管理工具使用相同的方式来对项目进行分类。对于版本号来说，应该让所有的工具都知道“1.1”是在“1.0”之后的。如果项目使用了自己定义的版本号命名方式，就无法做到这一点了。</p>

<p>要保证这种一致性，唯一的方法是让所有的项目都按照统一的方式来命名版本号。我们选择的方式是经典的序列版本号，在PEP 386中定义，它的格式是：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">N</span><span class="o">.</span><span class="n">N</span><span class="p">[</span><span class="o">.</span><span class="n">N</span><span class="p">]</span><span class="o">+</span><span class="p">[{</span><span class="n">a</span><span class="o">|</span><span class="n">b</span><span class="o">|</span><span class="n">c</span><span class="o">|</span><span class="n">rc</span><span class="p">}</span><span class="n">N</span><span class="p">[</span><span class="o">.</span><span class="n">N</span><span class="p">]</span><span class="o">+</span><span class="p">][</span><span class="o">.</span><span class="n">postN</span><span class="p">][</span><span class="o">.</span><span class="n">devN</span><span class="p">]</span>
</span></code></pre></td></tr></table></div></figure>


<p>其中：</p>

<ul>
<li><em>N</em> 是一个整数。你可以使用任意数量的N，用点号将它们分隔开来。但至少要有两个N，即“主版本.次版本”。</li>
<li><em>a, b, c</em> 分别是 <em>alpha, beta, release candidate</em> 的简写，它们后面还有一个整数。预发布版本有两种标记，c和rc，主要是为了和过去兼容，但c更简单些。</li>
<li><em>dev</em> 加一个数字表示开发版本。</li>
<li><em>post</em> 加一个数字表示已发布版本。</li>
</ul>


<p>根据项目发布周期的不同，开发版本和已发布版本可以作为两个最终版本之间的过渡版本号。大多数项目会使用开发版本。</p>

<p>按照这个形式，PEP 386定义了严格的顺序：</p>

<ul>
<li>alpha &lt; beta &lt; rc &lt; final</li>
<li>dev &lt; non-dev &lt; post, non-dev包括alpha, beta, rc或者final</li>
</ul>


<p>以下是一个完整的示例：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="mf">1.0</span><span class="n">a1</span> <span class="o">&lt;</span> <span class="mf">1.0</span><span class="n">a2</span><span class="o">.</span><span class="n">dev456</span> <span class="o">&lt;</span> <span class="mf">1.0</span><span class="n">a2</span> <span class="o">&lt;</span> <span class="mf">1.0</span><span class="n">a2</span><span class="o">.</span><span class="mf">1.</span><span class="n">dev456</span>
</span><span class='line'>  <span class="o">&lt;</span> <span class="mf">1.0</span><span class="n">a2</span><span class="o">.</span><span class="mi">1</span> <span class="o">&lt;</span> <span class="mf">1.0</span><span class="n">b1</span><span class="o">.</span><span class="n">dev456</span> <span class="o">&lt;</span> <span class="mf">1.0</span><span class="n">b2</span> <span class="o">&lt;</span> <span class="mf">1.0</span><span class="n">b2</span><span class="o">.</span><span class="n">post345</span>
</span><span class='line'>    <span class="o">&lt;</span> <span class="mf">1.0</span><span class="n">c1</span><span class="o">.</span><span class="n">dev456</span> <span class="o">&lt;</span> <span class="mf">1.0</span><span class="n">c1</span> <span class="o">&lt;</span> <span class="mf">1.0</span><span class="o">.</span><span class="n">dev456</span> <span class="o">&lt;</span> <span class="mf">1.0</span>
</span><span class='line'>      <span class="o">&lt;</span> <span class="mf">1.0</span><span class="o">.</span><span class="n">post456</span><span class="o">.</span><span class="n">dev34</span> <span class="o">&lt;</span> <span class="mf">1.0</span><span class="o">.</span><span class="n">post456</span>
</span></code></pre></td></tr></table></div></figure>


<p>这样定义的目标在于让其他打包系统能够将Python项目的版本号方便地转换成它们自己的版本命名规则。目前，如果上传的项目使用了PEP 345定义的元信息，PyPI会拒绝接受没有遵守PEP 386版本号命名规范的项目。</p>

<h4>依赖</h4>

<p>PEP 345定义了三个新的元信息属性，用来替换PEP 314中的<code>Requires</code>，<code>Provides</code>，和<code>Obsoletes</code>，它们是<code>Requires-Dist</code>，<code>Provides-Dist</code>，<code>Obsoletes-Dist</code>。这些属性可以在元信息中出现多次。</p>

<p><code>Requires-Dist</code>中定义了项目所依赖的软件包，使用依赖项目的<code>Name</code>元信息，并可以跟上一个版本号。这些依赖项目的名称必须能在PyPI中找到，且版本号命名规则要遵守PEP 386中的定义。以下是一些示例：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">Requires</span><span class="o">-</span><span class="n">Dist</span><span class="p">:</span> <span class="n">pkginfo</span>
</span><span class='line'><span class="n">Requires</span><span class="o">-</span><span class="n">Dist</span><span class="p">:</span> <span class="n">PasteDeploy</span>
</span><span class='line'><span class="n">Requires</span><span class="o">-</span><span class="n">Dist</span><span class="p">:</span> <span class="n">zope</span><span class="o">.</span><span class="n">interface</span> <span class="p">(</span><span class="o">&gt;</span><span class="mf">3.5</span><span class="o">.</span><span class="mi">0</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p><code>Provides-Dist</code>用来定义项目中包含的其他项目，常用于合并两个项目的情形。比如，ZODB项目可以包含名为<code>transaction</code>的项目，并声明：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">Provides</span><span class="o">-</span><span class="n">Dist</span><span class="p">:</span> <span class="n">transaction</span>
</span></code></pre></td></tr></table></div></figure>


<p><code>Obsoletes-Dist</code>主要用于将其它项目标记为本项目的过期版本。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">ObsoletesDist</span><span class="p">:</span> <span class="n">OldName</span>
</span></code></pre></td></tr></table></div></figure>


<h4>环境标识</h4>

<p>环境标识可以添加在上述三个属性的后面，使用分号分隔，用来标识该属性在什么样的目标环境中生效。以下是一些示例：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">Requires</span><span class="o">-</span><span class="n">Dist</span><span class="p">:</span> <span class="n">pywin32</span> <span class="p">(</span><span class="o">&gt;</span><span class="mf">1.0</span><span class="p">);</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span> <span class="o">==</span> <span class="s">&#39;win32&#39;</span>
</span><span class='line'><span class="n">Obsoletes</span><span class="o">-</span><span class="n">Dist</span><span class="p">:</span> <span class="n">pywin31</span><span class="p">;</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span> <span class="o">==</span> <span class="s">&#39;win32&#39;</span>
</span><span class='line'><span class="n">Requires</span><span class="o">-</span><span class="n">Dist</span><span class="p">:</span> <span class="n">foo</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="o">!=</span><span class="mf">1.3</span><span class="p">);</span> <span class="n">platform</span><span class="o">.</span><span class="n">machine</span> <span class="o">==</span> <span class="s">&#39;i386&#39;</span>
</span><span class='line'><span class="n">Requires</span><span class="o">-</span><span class="n">Dist</span><span class="p">:</span> <span class="n">bar</span><span class="p">;</span> <span class="n">python_version</span> <span class="o">==</span> <span class="s">&#39;2.4&#39;</span> <span class="ow">or</span> <span class="n">python_version</span> <span class="o">==</span> <span class="s">&#39;2.5&#39;</span>
</span><span class='line'><span class="n">Requires</span><span class="o">-</span><span class="n">External</span><span class="p">:</span> <span class="n">libxslt</span><span class="p">;</span> <span class="s">&#39;linux&#39;</span> <span class="ow">in</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span>
</span></code></pre></td></tr></table></div></figure>


<p>这种简易的语法足以让非Python程序员看懂：它使用<code>==</code>或<code>in</code>运算符（含<code>!=</code>和<code>not in</code>），且可以通过逻辑运算符连接。PEP 345中规定以下属性可以使用这种语法：</p>

<ul>
<li><code>Requires-Python</code></li>
<li><code>Requires-External</code></li>
<li><code>Requires-Dist</code></li>
<li><code>Provides-Dist</code></li>
<li><code>Obsoletes-Dist</code></li>
<li><code>Classifier</code></li>
</ul>


<h3>14.4.2 用户安装了什么？</h3>

<p>出于互通性的考虑，Python项目的安装格式必须一致。要让安装工具A能够检测到工具B安装的项目，它们就必须共享和更新相同的项目列表。</p>

<p>当然，理想中用户会在系统中只使用一种安装工具，但是他们也许会需要迁移到另一种工具以获得一些新的特性。比如，Mac OS X操作系统自带了<code>Setuptools</code>，因而装有<code>easy_install</code>工具。当他们想要切换到新的工具时，该工具就必须兼容现有的环境。</p>

<p>如果系统使用类似RPM这样的工具管理Python软件包，那么其它安装工具在安装新项目时是无法通知到系统的。更糟糕的是，即便Python安装工具能够通知到中央打包系统，我们也必须在Python元信息和系统元信息之间做一个映射。比如，项目的名称在两个系统中可能是不一致的。造成这种问题的原因也多种多样，比较常见的原因是命名冲突，即RPM源中已经有一个同名的项目了。另一个原因是项目名称中包含了<code>python</code>这个前缀，从而破坏了RPM系统的规范。比如，你的项目名称是<code>foo-python</code>，那在RPM源中很可能被表示为<code>python-foo</code>。</p>

<p>一种解决办法是不去触碰全局的Python环境，而是使用一个隔离的环境，如<code>Virtualenv</code>。</p>

<p>但不管怎样，采用统一的Python安装格式还是有必要的，因为其它一些打包系统在为自己安装Python项目时还是需要考虑互通性。当第三方打包系统新安装了一个项目，并在自身的数据库中注册后，它还需要为Python安装环境生成一个正确的元信息，从而让项目在这个环境中变得可见，或能通过该Python环境提供的API检索到。</p>

<p>元信息的映射问题可以这样描述：因为RPM系统知道自己安装了哪些Python项目，它就能生成合适的Python元信息。例如，它知道<code>python26-webob</code>项目在PyPI中的名字是<code>WebOb</code>。</p>

<p>回到我们的规范：PEP 376定义的项目安装规范和<code>Seteptools</code>以及<code>Pip</code>的格式很相似，它是一个以<code>dist-info</code>结尾的目录，包含以下内容：</p>

<ul>
<li><code>METADATA</code>：元信息，其格式在PEP 345、PEP 314和PEP 241中描述。</li>
<li><code>RECORD</code>：项目安装的文件列表，以类似csv的格式保存。</li>
<li><code>INSTALLER</code>：安装项目所使用的工具。</li>
<li><code>REQUESTED</code>：如果这个文件存在，则表明这个项目是被显式安装的，即并不是作为依赖项而安装。</li>
</ul>


<p>如果所有的安装工具都能识别这种格式，我们在管理Python项目时就不需要依赖特定的安装工具和它提供的特性了。此外，PEP 376将元信息设计为一个目录，这样就能方便地扩展。事实上，下一章要描述的<code>RESOURCES</code>文件很可能会在不久的将来添加到元信息中，而不用改变PEP 376标准。当事实证明这个文件能被所有的安装工具使用，则会将它修订到PEP中。</p>

<h3>14.4.3 数据文件的结构</h3>

<p>前面已经提到，我们需要能够让打包者来决定项目的数据文件安装在哪个位置，而不用修改代码。同样，也要能够让开发者在开发时不用去考虑数据文件的存放位置。我们的解决方案很普通：重定向。</p>

<h4>使用数据文件</h4>

<p>假设你的<code>MPTools</code>项目需要使用一个配置文件。开发者会将改文件放到Python包安装目录中，并使用<code>__file__</code>去引用：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">import</span> <span class="nn">os</span>
</span><span class='line'>
</span><span class='line'><span class="n">here</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">__file__</span><span class="p">)</span>
</span><span class='line'><span class="n">cfg</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">here</span><span class="p">,</span> <span class="s">&#39;config&#39;</span><span class="p">,</span> <span class="s">&#39;mopy.cfg&#39;</span><span class="p">))</span>
</span></code></pre></td></tr></table></div></figure>


<p>这样编写代码意味着该配置文件必须和代码放在相同的位置，一个名为<code>config</code>的子目录下。</p>

<p>我们设计的新的数据文件结构以项目为根节点，开发者可以定义任意的文件目录结构，而不用关心根目录是存放在软件安装目录中或是其它目录。开发者可以使用<code>pkgutil.open</code>来访问这些数据文件：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">import</span> <span class="nn">os</span>
</span><span class='line'><span class="kn">import</span> <span class="nn">pkgutil</span>
</span><span class='line'>
</span><span class='line'><span class="c"># Open the file located in config/mopy.cfg in the MPTools project</span>
</span><span class='line'><span class="n">cfg</span> <span class="o">=</span> <span class="n">pkgutil</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="s">&#39;MPTools&#39;</span><span class="p">,</span> <span class="s">&#39;config/mopy.cfg&#39;</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p><code>pkgutil.open</code>命令会检索项目元信息中的<code>RESOURCES</code>文件，该文件保存的是一个简单的映射信息——文件名称和它所存放的位置：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">config</span><span class="o">/</span><span class="n">mopy</span><span class="o">.</span><span class="n">cfg</span> <span class="p">{</span><span class="n">confdir</span><span class="p">}</span><span class="o">/</span><span class="p">{</span><span class="n">distribution</span><span class="o">.</span><span class="n">name</span><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>其中，<code>{confdir}</code>变量指向系统的配置文件目录，<code>{distribution.name}</code>变量表示的是Python项目名称。</p>

<p><img src="http://www.aosabook.org/images/packaging/find-file.png" alt="图14.4：定位一个文件" /></p>

<p>图14.4：定位一个文件</p>

<p>只要安装过程中生成了<code>RESOURCES</code>文件，这个API就能帮助开发者找到<code>mopy.cfg</code>文件。又因为<code>config/mopy.cfg</code>是一个相对于项目的路径，我们就能在开发模式下提供一个本地的路径，让<code>pkgutil</code>能够找到它。</p>

<h4>声明数据文件</h4>

<p>实际使用中，我们可以在<code>setup.cfg</code>文件中用映射关系来定义数据文件的存放位置。映射关系的形式是<code>(glob-style pattern, target)</code>，每个“模式”指向项目中的一个或一组文件，“目标”则表示实际安装位置，可以包含变量名，用花括号括起。例如，<code>MPTools</code>的<code>setup.cfg</code>文件可以是以下内容：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="p">[</span><span class="n">files</span><span class="p">]</span>
</span><span class='line'><span class="n">resources</span> <span class="o">=</span>
</span><span class='line'>        <span class="n">config</span><span class="o">/</span><span class="n">mopy</span><span class="o">.</span><span class="n">cfg</span> <span class="p">{</span><span class="n">confdir</span><span class="p">}</span><span class="o">/</span><span class="p">{</span><span class="n">application</span><span class="o">.</span><span class="n">name</span><span class="p">}</span><span class="o">/</span>
</span><span class='line'>        <span class="n">images</span><span class="o">/*.</span><span class="n">jpg</span>    <span class="p">{</span><span class="n">datadir</span><span class="p">}</span><span class="o">/</span><span class="p">{</span><span class="n">application</span><span class="o">.</span><span class="n">name</span><span class="p">}</span><span class="o">/</span>
</span></code></pre></td></tr></table></div></figure>


<p><code>sysconfig</code>模块提供了一组可用的变量，并为不同的操作系统提供了默认值。例如，<code>{confdir}</code>在Linux下是<code>/etc</code>。安装工具就能结合<code>sysconfig</code>模块来决定数据文件的存放位置。最后，它会生成一个<code>RESOURCES</code>文件，这样<code>pkgutil</code>就能找到这些文件了：</p>

<p><img src="http://www.aosabook.org/images/packaging/installer.png" alt="图14.5：安装工具" /></p>

<p>图14.5：安装工具</p>

<h3>14.4.4 改进PypI</h3>

<p>上文提到过，PyPI目前是一个单点故障源。PEP 380中正式提出了这个问题，并定义了一个镜像协议，使得用户可以在PyPI出现问题时连接到其他源。这个协议的目的是让社区成员可以在世界各地搭建起PyPI镜像。</p>

<p><img src="http://www.aosabook.org/images/packaging/mirroring.png" alt="图14.6：镜像" /></p>

<p>图14.6：镜像</p>

<p>镜像列表的格式是<code>X.pypi.python.org</code>，其中<code>X</code>是一个字母序列，如<code>a,b,c,…,aa,ab,….</code>，<code>a.pypi.python.org</code>是主服务器，b字母开始是从服务器。域名<code>last.pypi.python.org</code>的A记录指向这个列表中的最后一个服务器，这样PyPI的使用者就能够根据DNS记录来获取完整的服务器镜像列表了。</p>

<p>比如，以下代码获取到的最后一个镜像地址是<code>h.pypi.python.org</code>，表示当前PyPI有7个镜像服务器（b至h）：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">socket</span>
</span><span class='line'><span class="o">&gt;&gt;&gt;</span> <span class="n">socket</span><span class="o">.</span><span class="n">gethostbyname_ex</span><span class="p">(</span><span class="s">&#39;last.pypi.python.org&#39;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
</span><span class='line'><span class="s">&#39;h.pypi.python.org&#39;</span>
</span></code></pre></td></tr></table></div></figure>


<p>这样一来，客户端还可以根据域名的IP地址来决定连接最近的镜像服务器，或者在服务器发生故障时自动重连到新的地址。镜像协议本身要比rsync更复杂一些，因为我们需要保证下载统计量的准确性，并提供最基本的安全性保障。</p>

<h4>同步</h4>

<p>镜像必须尽可能降低和主服务器之间的数据交换，要达到这个目的，就必须在PyPI的XML-RPC接口中加入<code>changelog</code>信息，以保证只获取变化的内容。对于每个软件包“P”，镜像必须复制<code>/simple/P/</code>和<code>/serversig/P</code>这两组信息。</p>

<p>如果中央服务器中删除了一个软件包，它就必须删除所有和它有关的数据。为了检测软件包文件的变动，可以缓存文件的ETag信息，并通过<code>If-None-Match</code>头来判断是否可以跳过传输过程。当同步完成后，镜像就将<code>/last-modified</code>文件设置为当前的时间。</p>

<h4>统计信息</h4>

<p>当用户在镜像中下载一个软件包时，镜像就需要将这个事件报告给中央服务器，继而广播给其他镜像服务器。这样就能保证下载工具在任意镜像都能获得正确的下载量统计信息。</p>

<p>统计信息以CSV文件的格式保存在中央服务器的<code>stats</code>目录中，按照日和周分隔。每个镜像服务器需要提供一个<code>local-stats</code>目录来存放它自己的统计信息。文件中保存了每个软件包的下载数量，以及它们的下载工具。中央服务器每天都会从镜像服务器中获取这些信息，将其合并到全局的<code>stats</code>目录，这样就能保证镜像服务器中的<code>local-stats</code>目录中的数据至少是每日更新的。</p>

<h4>镜像服务器的合法性</h4>

<p>在分布式的镜像系统中，客户端需要能够验证镜像服务器的合法性。如果不这样做，就可能产生以下威胁：</p>

<ul>
<li>中央索引发生错误</li>
<li>镜像服务被篡改</li>
<li>服务器和客户端之间遭到拦截攻击</li>
</ul>


<p>对于第一种攻击，软件包的作者就需要使用自己的PGP密钥来对软件包进行加密，这样其他用户就能判断他所下载的软件包是来自可信任的作者的。镜像服务协议中只对第二种攻击做了预防，不过有些措施也可以预防拦截攻击。</p>

<p>中央服务器会在<code>/serverkey</code>这个URL下提供一个DSA密钥，它是用<code>opensll dsa-pubout</code><sup>3</sup>生成的PEM格式的密钥。这个URL不能被镜像服务器收录，客户端必须从主服务器中获取这个serverkey密钥，或者使用PyPI客户端本身自带的密钥。镜像服务器也是需要下载这个密钥的，用来检测密钥是否有更新。</p>

<p>对于每个软件包，<code>/serversig/package</code>中存放了它们的镜像签名。这是一个DSA签名，和URL<code>/simple/package</code>包含的内容对等，采用DER格式，是SHA-1和DSA的结合<sup>4</sup>。</p>

<p>客户端从镜像服务器下载软件包时必须经过以下验证：</p>

<ol>
<li>下载<code>/simple</code>页面，计算它的<code>SHA-1</code>哈希值。</li>
<li>计算这个哈希值的DSA签名。</li>
<li>下载对应的<code>/serversig</code>，将它和第二步中生成的签名进行比对。</li>
<li>计算并验证所有下载文件的MD5值（和<code>/simple</code>页面中的内容对比）。</li>
</ol>


<p>在从中央服务器下载软件包时不需要进行上述验证，客户端也不应该进行验证，以减少计算量。</p>

<p>这些密钥大约每隔一年会被更新一次。镜像服务器需要重新获取所有的<code>/serversig</code>页面内容，使用镜像服务的客户端也需要通过可靠的方式获取新密钥。一种做法是从<code>https://pypi.python.org/serverkey</code>下载。为了检测拦截攻击，客户端需要通过CAC认证中心验证服务端的SSL证书。</p>

<h2>14.5 实施细节</h2>

<p>上文提到的大多数改进方案都在<code>Distutils2</code>中实现了。<code>setup.py</code>文件已经退出历史舞台，取而代之的是<code>setup.cfg</code>，一个类似<code>.ini</code>类型的文件，它描述了项目的所有信息。这样做可以让打包人员方便地改变软件包的安装方式，而不需要接触Python语言。以下是一个配置文件的示例：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="p">[</span><span class="n">metadata</span><span class="p">]</span>
</span><span class='line'><span class="n">name</span> <span class="o">=</span> <span class="n">MPTools</span>
</span><span class='line'><span class="n">version</span> <span class="o">=</span> <span class="mf">0.1</span>
</span><span class='line'><span class="n">author</span> <span class="o">=</span> <span class="n">Tarek</span> <span class="n">Ziade</span>
</span><span class='line'><span class="n">author</span><span class="o">-</span><span class="n">email</span> <span class="o">=</span> <span class="n">tarek</span><span class="nd">@mozilla.com</span>
</span><span class='line'><span class="n">summary</span> <span class="o">=</span> <span class="n">Set</span> <span class="n">of</span> <span class="n">tools</span> <span class="n">to</span> <span class="n">build</span> <span class="n">Mozilla</span> <span class="n">Services</span> <span class="n">apps</span>
</span><span class='line'><span class="n">description</span><span class="o">-</span><span class="nb">file</span> <span class="o">=</span> <span class="n">README</span>
</span><span class='line'><span class="n">home</span><span class="o">-</span><span class="n">page</span> <span class="o">=</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">bitbucket</span><span class="o">.</span><span class="n">org</span><span class="o">/</span><span class="n">tarek</span><span class="o">/</span><span class="n">pypi2rpm</span>
</span><span class='line'><span class="n">project</span><span class="o">-</span><span class="n">url</span><span class="p">:</span> <span class="n">Repository</span><span class="p">,</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">hg</span><span class="o">.</span><span class="n">mozilla</span><span class="o">.</span><span class="n">org</span><span class="o">/</span><span class="n">services</span><span class="o">/</span><span class="n">server</span><span class="o">-</span><span class="n">devtools</span>
</span><span class='line'><span class="n">classifier</span> <span class="o">=</span> <span class="n">Development</span> <span class="n">Status</span> <span class="p">::</span> <span class="mi">3</span> <span class="o">-</span> <span class="n">Alpha</span>
</span><span class='line'>    <span class="n">License</span> <span class="p">::</span> <span class="n">OSI</span> <span class="n">Approved</span> <span class="p">::</span> <span class="n">Mozilla</span> <span class="n">Public</span> <span class="n">License</span> <span class="mf">1.1</span> <span class="p">(</span><span class="n">MPL</span> <span class="mf">1.1</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="p">[</span><span class="n">files</span><span class="p">]</span>
</span><span class='line'><span class="n">packages</span> <span class="o">=</span>
</span><span class='line'>        <span class="n">mopytools</span>
</span><span class='line'>        <span class="n">mopytools</span><span class="o">.</span><span class="n">tests</span>
</span><span class='line'>
</span><span class='line'><span class="n">extra_files</span> <span class="o">=</span>
</span><span class='line'>        <span class="n">setup</span><span class="o">.</span><span class="n">py</span>
</span><span class='line'>        <span class="n">README</span>
</span><span class='line'>        <span class="n">build</span><span class="o">.</span><span class="n">py</span>
</span><span class='line'>        <span class="n">_build</span><span class="o">.</span><span class="n">py</span>
</span><span class='line'>
</span><span class='line'><span class="n">resources</span> <span class="o">=</span>
</span><span class='line'>    <span class="n">etc</span><span class="o">/</span><span class="n">mopytools</span><span class="o">.</span><span class="n">cfg</span> <span class="p">{</span><span class="n">confdir</span><span class="p">}</span><span class="o">/</span><span class="n">mopytools</span>
</span></code></pre></td></tr></table></div></figure>


<p><code>Distutils2</code>会将这个文件用作于：</p>

<ul>
<li>生成<code>META-1.2</code>格式的元信息，可以用作多种用途，如在PyPI上注册项目。</li>
<li>执行任何打包管理命令，如<code>sdist</code>。</li>
<li>安装一个以<code>Distutils2</code>为基础的项目。</li>
</ul>


<p><code>Distutils2</code>还通过<code>version</code>模块实现了<code>VERSION</code>元信息。</p>

<p>对<code>INSTALL-DB</code>元信息的实现会被包含在Python3.3的<code>pkgutil</code>模块中。在过度版本中，它的功能会由<code>Distutils2</code>完成。它所提供的API可以让我们浏览系统中已安装的项目。</p>

<p>以下是<code>Distutils2</code>提供的核心功能：</p>

<ul>
<li>安装、卸载</li>
<li>依赖树</li>
</ul>


<h2>14.6 经验教训</h2>

<h3>14.6.1 PEP的重要性</h3>

<p>要改变像Python打包系统这样庞大和复杂的架构必须通过谨慎地修改PEP标准来进行。据我所知，任何对PEP的修改和添加都要历经一年左右的时间。</p>

<p>社区中一直以来有个错误的做法：为了改善某个问题，就肆意扩展项目元信息，或是修改Python程序的安装方式，而不去尝试修订它所违背的PEP标准。</p>

<p>换句话说，根据你所使用的安装工具的不同，如<code>Distutils</code>和<code>Setuptools</code>，它们安装应用程序的方式就是不同的。这些工具的确解决了一些问题，但却会引发一连串的新问题。以操作系统的打包工具为例，管理员必须面对多个Python标准：官方文档所描述的标准，<code>Setuptools</code>强加给大家的标准。</p>

<p>但是，<code>Setuptools</code>能够有机会在实际环境中大范围地（在整个社区中）进行实验，创新的进度很快，得到的反馈信息也是无价的。我们可以据此撰写出更切合实际的PEP新标准。所以，很多时候我们需要能够察觉到某个第三方工具在为Python社区做出贡献，并应该起草一个新的PEP标准来解决它所提出的问题。</p>

<h3>14.6.2 一个被纳入标准库的项目就已经死亡了一半</h3>

<p>这个标题是援引Guido van Rossum的话，而事实上，Python的这种战争式的哲学也的确冲击了我们的努力成果。</p>

<p><code>Distutils</code>是Python标准库之一，将来<code>Distutils2</code>也会成为标准库。一个被纳入标准库的项目很难再对其进行改造。虽然我们有正常的项目更新流程，即经过两个Python次版本就可以对某个API进行删改，但一旦某个API被发布，它必定会持续存在多年。</p>

<p>因此，对标准库中某个项目的一次修改并不是简单的bug修复，而很有可能影响整个生态系统。所以，当你需要进行重大更新时，就必须创建一个新的项目。</p>

<p>我之所以深有体会，就是因为在我对<code>Distutils</code>进行了超过一年的修改后，还是不得不回滚所有的代码，开启一个新的<code>Distutils2</code>项目。将来，如果我们的标准又一次发生了重大改变，很有可能会产生<code>Distutils3</code>项目，除非未来某一天标准库会作为独立的项目发行。</p>

<h3>14.6.3 向前兼容</h3>

<p>要改变Python项目的打包方式，其过程是非常漫长的：Python的生态系统中包含了那么多的项目，它们都采用旧的打包工具管理，一定会遇到诸多阻力。（文中一些章节描述的问题，我们花费了好几年才达成共识，而不是我之前预想的几个月。）对于Python3，可能会花费数年的时间才能将所有的项目都迁移到新的标准中去。</p>

<p>这也是为什么我们做的任何修改都必须兼容旧的打包工具，这是<code>Distutils2</code>编写过程中非常棘手的问题。</p>

<p>例如，一个以新标准进行打包的项目可能会依赖一个尚未采用新标准的其它项目，我们不能因此中断安装过程，并告知用户这是一个无法识别的依赖项。</p>

<p>举例来说，<code>INSTALL-DB</code>元信息的实现中会包含那些用<code>Distutils</code>、<code>Pip</code>、<code>Distribution</code>、或<code>Setuptools</code>安装的项目。<code>Distutils2</code>也会为那些使用<code>Distutils</code>安装的项目生成新的元信息。</p>

<h2>14.7 参考和贡献者</h2>

<p>本文的部分章节直接摘自PEP文档，你可以在<code>http://python.org</code>中找到原文：</p>

<ul>
<li>PEP 241: Metadata for Python Software Packages 1.0: http://python.org/peps/pep-0214.html</li>
<li>PEP 314: Metadata for Python Software Packages 1.1: http://python.org/peps/pep-0314.html</li>
<li>PEP 345: Metadata for Python Software Packages 1.2: http://python.org/peps/pep-0345.html</li>
<li>PEP 376: Database of Installed Python Distributions: http://python.org/peps/pep-0376.html</li>
<li>PEP 381: Mirroring infrastructure for PyPI: http://python.org/peps/pep-0381.html</li>
<li>PEP 386: Changing the version comparison module in Distutils: http://python.org/peps/pep-0386.html</li>
</ul>


<p>在这里我想感谢所有为打包标准的制定做出贡献的人们，你可以在PEP中找到他们的名字。我还要特别感谢“打包别动队”的成员们。还要谢谢Alexis Metaireau、Toshio Kuratomi、Holger Krekel、以及Stefane Fermigier，感谢他们对本文提供的反馈。</p>

<p>本章中讨论的项目有：</p>

<ul>
<li>Distutils: http://docs.python.org/distutils</li>
<li>Distutils2: http://packages.python.org/Distutils2</li>
<li>Distribute: http://packages.python.org/distribute</li>
<li>Setuptools: http://pypi.python.org/pypi/setuptools</li>
<li>Pip: http://pypi.python.org/pypi/pip</li>
<li>Virtualenv: http://pypi.python.org/pypi/virtualenv</li>
</ul>


<h2>脚注</h2>

<ol>
<li>文中引用的Python改进提案（Python Enhancement Proposals，简称PEP）会在本文最后一节整理。</li>
<li>过去被命名为CheeseShop</li>
<li>即RFC 3280 SubjectPublishKeyInfo中定义的1.3.14.3.2.12算法。</li>
<li>即RFC 3279 Dsa-Sig-Value中定义的1.2.840.10040.4.3算法。</li>
</ol>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[开源软件架构 - 卷2：第6章 Git]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/11/16/aosa-git/"/>
    <updated>2012-11-16T23:54:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/11/16/aosa-git</id>
    <content type="html"><![CDATA[<p>作者：<a href="http://www.aosabook.org/en/intro2.html#potter-susan">Susan Potter</a>，翻译：<a href="mailto:zhangji87@gmail.com">张吉</a></p>

<p>原文：<a href="http://www.aosabook.org/en/git.html">http://www.aosabook.org/en/git.html</a></p>

<h2>6.1 Git概述</h2>

<p>Git能够让不同的协作者通过一个点对点的仓库网络对数据内容（通常是代码，当然不仅限于代码）进行维护。它支持分布式的工作流程，能够让数据内容临时分离，并最终合并到一起。</p>

<p>本章将阐述Git的内部实现是如何提供以上功能的，以及它和其他版本控制系统（VCS）的区别。</p>

<h2>6.2 Git起源</h2>

<p>为了更好地理解Git的设计思想，我们有必要先了解一下Git项目的发源地——Linux内核开发社区——所面临的问题。</p>

<p>Linux内核开发与其他商业软件项目有很大不同，因为它的开发者众多，且每个开发者的参与程度和对Linux内核代码的理解有很大差异。多年以来，内核代码一直都是以Tar压缩文件以及补丁的形式维护的，而当时的核心开发团队一直在寻找一个能够满足他们各方面需求的版本控制系统。</p>

<p>Git就是在这样的背景下于2005年作为一款开源软件诞生的。当时，Linux内核代码通过两种版本控制系统进行维护，BitKeeper和CVS，分别由两组核心开发团队使用。BitKeeper相较于当时颇为流行的CVS，提供了一种不同的历史展示方式。</p>

<!-- more -->


<p>当BitKeeper的所有者BitMover决定收回Linux内核开发人员的使用许可时，Linux Torvalds紧急开启了一个项目，也就是后来的Git。一开始，他通过编写一组Shell脚本来帮助他将邮件中的补丁按顺序应用到代码中。这组原始脚本能够在代码合并过程中迅速中断，让维护者能够进行人工干预，修改代码，然后继续合并。</p>

<p>从项目开始之初，Torvalds就为Git制定了一个目标——要和CVS的做法完全相反——同时还包含了以下三条设计目标：</p>

<ul>
<li>支持分布式的协作流程，类似BitKeeper</li>
<li>预防代码错乱</li>
<li>高性能</li>
</ul>


<p>这些设计目标都被实现了，我会在下文中通过解析Git的各种做法来阐述，包括在内容管理中使用有向无环图（<a href="http://en.wikipedia.org/wiki/Directed_acyclic_graph">DAG</a>），头指针引用，对象模型，远程协议，以及Git如何追踪合并树。</p>

<p>虽然Git设计之初受到了很多BitKeeper的影响，但是两者还是有根本上的区别的，如Git提供了更多分布式和本地开发流程，这点是BitKeeper做不到的。<a href="http://www.monotone.ca/">Monotone</a>，2003年启动的一个开源分布式版本控制系统，也对Git的早期开发产生了影响。</p>

<p>分布式版本控制系统在提供更灵活的工作流程的同时，往往会增加它的复杂程度。分布式模型的独特优点有：</p>

<ul>
<li>能够线下进行增量提交</li>
<li>开发者可以决定自己的代码何时能够开放出来</li>
<li>能够线下浏览历史</li>
<li>可以将工作成果发布到不同的仓库，以不同的分支、不同的提交粒度展现出来</li>
</ul>


<p>在Git项目的开发期间，诞生了其他三个开源分布式版本控制系统（其中Mercurial可以参见《开源软件架构》的第一卷）。这些分布式版本控制系统（dVCS）都提供了非常灵活的工作流程，这是先前的集中式版本控制系统做不到的。注意：Subversion有一款插件名为SVK，由不同的开发者维护，提供了服务器之间的同步功能。</p>

<p>目前流行的dVCS包括Bazaar, Darcs, Fossil, Git, Mercurial, 以及Veracity。</p>

<h2>6.3 版本控制系统的设计</h2>

<p>现在让我们回过头来看看Git之外的其他版本控制系统是如何设计的。通过比较他们和Git之间的区别，可以帮助我们去理解Git在架构设计中的选择。</p>

<p>版本控制系统通常有三项核心功能（需求）：</p>

<ul>
<li>保存内容</li>
<li>记录变更历史（包括具体的合并信息）</li>
<li>向协作者分发内容和变更历史</li>
</ul>


<p>注意：第三项并不是所有版本控制系统的核心功能。</p>

<h3>保存内容</h3>

<p>在VCS中保存内容，最普遍的做法是保存增量的修改，或使用有向无环图（DAG）。</p>

<p>增量修改可以反映出两个版本之间的内容差异，以及一些额外的信息。使用有向无环图保存内容则是将特定对象构造成一种树状结构，作为某一次提交的快照保存下来（树状结构中未发生变化的对象是可以重用的）。Git使用有向无环图来保存内容，它所使用的不同对象类型会在本文的“对象数据库”一节中有所描述。</p>

<h3>提交和合并的历史</h3>

<p>在保存历史、记录变化方面，大部分VCS使用以下方式之一：</p>

<ul>
<li>线性历史</li>
<li>有向无环图</li>
</ul>


<p>Git使用的还是有向无环图，这次则是用来保存历史。每次提交包含了它父节点的元信息——Git中的一次提交可以拥有0个或多个父节点（理论上没有个数限制）。例如，Git仓库的第一次提交就没有父节点，而一次三头合并则有三个父节点。</p>

<p>Git和SVN线性提交的另一个重要区别是Git可以直接进行分支的创建，并记录下大部分合并历史。</p>

<p><img src="http://www.aosabook.org/images/git/dag-example.png" alt="图6.1：Git中有向无环图示例" /></p>

<p>图6.1：Git中有向无环图示例</p>

<p>通过采用有向无环图保存内容，Git能够提供完整的分支功能。一个文件的历史会通过它所处的目录结构位置和根节点关联起来，并最终和一个提交节点关联。这个提交节点又会有一个或多个父节点。这种组织方式提供了以下两个特性，让我们能够更好地在Git浏览文件历史和内容：</p>

<ul>
<li>当内容节点（文件或目录）在有向无环图中有相同的标识（Git中以SHA码表示），即使它们处于不同的提交节点，也能保证它们的内容是一致的，从而使得Git在差异比对时更为高效。</li>
<li>在对两个分支进行合并时，实质上是在对两个有向无环图节点进行合并。有向无环图能够让Git更为高效地判断出他们共同的父节点。</li>
</ul>


<h3>内容分发</h3>

<p>版本控制系统在向协作者分发内容时通常有以下三种做法：</p>

<ul>
<li>仅限本地：某些版本控制系统没有上文提到的第三项需求。</li>
<li>中央服务器：版本库的所有改动都必须在一个中央版本库中进行，也只有这个版本库会记录历史。</li>
<li>分布式模型：虽然分布式模型中也会有一个中央仓库供协作者“推送”自己的改动，但协作者可以在本地进行提交，并稍后再推送到远程。</li>
</ul>


<p>为了展示以上设计模式的优点和不足，我们设想这样一个应用场景：一个SVN仓库和一个Git仓库，有着相同的内容（即Git默认分支的头指针指向的内容和SVN仓库最新的trunk分支内容一致）。一个名叫Alex的开发者在本地检出了一份SVN代码，以及克隆了一个Git版本库。</p>

<p>假设Alex在本地对一个1M大小的文件进行了修改，并进行了提交。提交后本地更新了元信息，远程服务器则是将文件的差异记录了下来。</p>

<p>Git下则有所不同。Alex对文件的变动首先会在本地进行记录，然后再“推送”到远程的公共仓库，这样文件的改动就能被其他开发者看到了。文件内容的变动记录在不同的版本库之中的表示方式是完全一致的。除了本地提交之外，Git会为变动后的文件创建一个对象来保存它（包括其完整的内容），然后逐层为该文件的父目录创建对象，直至仓库根目录。接下来Git会创建一个有向无环图，从刚才新创建的根目录节点开始，指向各个二进制单元（期间会重用那些内容没有改变的二进制单元），并使用新创建的二进制单元去替代那些变动的部分（一个二进制单元通常用来表示一个文件）。</p>

<p>到此为止，本次提交还是只保存在Alex克隆下来的本地仓库中。当Alex将这个提交推送到远程仓库后，远程仓库会验证这次提交是否能应用到当前分支中，然后这些对象将会按照原样保存下来，如同在本地仓库中创建的一样。</p>

<p>在Git中会有很多可变动的部分，有些对用户是透明的，有些则需要用户显示地指定这些内容是否需要分享出来，或是只在本地保存。虽然增加了复杂性，但也提供给团队开发者更大的自由度，得以更好地控制工作流程和发布内容，这在“Git起源”一节中已经有所阐述。</p>

<p>在SVN中，开发者不会忘记将变动内容提交至远程仓库。从效率上讲，SVN仅保存变动内容的方式会比Git保存文件每个版本的完整内容要来得高效，但是之后我们会讲述Git其实已经通过某种方式对此进行了优化。</p>

<h2>6.4 工具包</h2>

<p>如今，Git已然形成一个生态系统，在各种操作系统上（包括Windows）都开发出了大量命令行和图界面工具，而他们大部分都是构建在Git核心工具包之上的。</p>

<p>由于Git是Linus发起和开发的，它又立足于Linux社区，因此Git工具包的设计理念和传统的Unix命令行工具相仿。</p>

<p>Git工具包分为两个部分：底层命令和上层命令。底层命令提供了基本的内容追踪手段，以及直接操纵有向无环图。上层命令则是用户主要接触的命令，用以维护仓库，以及在多个仓库间进行协作。</p>

<p>虽然Git工具包提供了足够多的命令来操纵仓库，但是开发者们还是抱怨Git没有提供类库以供调用。Git命令最终会执行die()方法，使得GUI和Web界面在使用它时必须启动一个新的进程，效率较低。</p>

<p>不过这一问题已经得到处理，我会在本文的“当前进展和未来规划”一节加以阐述。</p>

<h2>6.5 版本库、暂存区、工作区</h2>

<p>让我们开始深入研究一下Git吧，了解其中几个关键概念。</p>

<p>首先让我们在本地创建一个Git版本库。在类Unix系统下，我们可以执行以下命令：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>mkdir testgit
</span><span class='line'><span class="nv">$ </span><span class="nb">cd </span>testgit
</span><span class='line'><span class="nv">$ </span>git init
</span></code></pre></td></tr></table></div></figure>


<p>这样我们就在testgit目录中初始化了一个新的版本库。我们可以建立分支、提交、创建里程碑、和远程Git仓库进行交互。我们甚至可以和其他类型的版本控制系统进行交互，只需要借助若干<code>git</code>命令即可。</p>

<p><code>git init</code>命令会在testgit目录下创建一个名为.git的子目录。我们来看一下这个目录的结构：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>tree .git/
</span><span class='line'>.git/
</span><span class='line'>|-- HEAD
</span><span class='line'>|-- config
</span><span class='line'>|-- description
</span><span class='line'>|-- hooks
</span><span class='line'>|   |-- applypatch-msg.sample
</span><span class='line'>|   |-- commit-msg.sample
</span><span class='line'>|   |-- post-commit.sample
</span><span class='line'>|   |-- post-receive.sample
</span><span class='line'>|   |-- post-update.sample
</span><span class='line'>|   |-- pre-applypatch.sample
</span><span class='line'>|   |-- pre-commit.sample
</span><span class='line'>|   |-- pre-rebase.sample
</span><span class='line'>|   |-- prepare-commit-msg.sample
</span><span class='line'>|   |-- update.sample
</span><span class='line'>|-- info
</span><span class='line'>|   |-- exclude
</span><span class='line'>|-- objects
</span><span class='line'>|   |-- info
</span><span class='line'>|   |-- pack
</span><span class='line'>|-- refs
</span><span class='line'>    |-- heads
</span><span class='line'>    |-- tags
</span></code></pre></td></tr></table></div></figure>


<p><code>.git</code>目录默认创建在工作区的根目录下，也就是<code>testgit</code>。它包含了以下几种类型的文件和目录：</p>

<ul>
<li><em>配置文件</em>： <code>.git/config</code>、<code>.git/description</code>、<code>.git/info/exclude</code>，这些文件会用来配置本地仓库。</li>
<li><em>钩子</em>： <code>.git/hooks</code>目录下的脚本可以在Git运作的各个环节中得到执行。</li>
<li><em>暂存区</em>： <code>.git/index</code>文件（它并没有在上述目录结构中显示出来）会用来保存工作区准备提交的内容。</li>
<li><em>对象数据库</em>： <code>.git/objects</code>是默认的Git对象数据库存放目录，囊括了本地仓库的所有文件内容和指针。对象一经创建则不能修改。</li>
<li><em>引用</em>：<code>.git/refs</code>目录用来存放本地和远程仓库的分支、里程碑、头指针等信息。“引用”表示指向某个对象指针，通常是<code>tag</code>和<code>commit</code>类型。引用之所以放置在对象数据库之外，是为了让他们能够随版本库的演进而变化。特殊的引用可以指向其他引用，如<code>HEAD</code>。</li>
</ul>


<p><code>.git</code>目录是真正意义上的版本库。工作区指的是包含所有工作文件的目录，它通常是<code>.git</code>目录的父目录。如果你需要创建一个没有工作区的远程仓库，可以使用<code>git init --bare</code>命令。它会直接在根目录下生成Git仓库的各类文件，而不是放置在一个子目录中。</p>

<p>另一个较为重要的文件是Git暂存区：<code>.git/index</code>。它在工作区和本地版本库之间增加了一个缓冲区，可以将需要提交的内容暂存在这里，最后一起提交。即使你对很多文件进行了修改，通过暂存区可以将它们作为一次完整的提交，并加注合理的注释。如果想将工作区某些文件的部分修改保存至暂存区，可以使用<code>git add -p</code>命令。</p>

<p>Git暂存区里的内容默认保存在单个文件中。版本库、暂存区、工作区的存放位置都是可以通过环境变量来进行配置的。</p>

<p>我们有必要了解一下以上三个区域的文件是如何进行交互的，以几个核心的Git命令举例：</p>

<ul>
<li><p><code>git checkout [branch]</code></p>

<p>  这条命令会将HEAD引用指向指定分支的引用（如<code>refs/heads/master</code>），并用该引用指向的内容替换掉暂存区和工作区中的内容。</p></li>
<li><p><code>git add [files]</code></p>

<p>  这条命令会检验工作区中指定的文件和暂存区是否一致，若不一致则更新暂存区。版本库不会发生变化。</p></li>
</ul>


<p>为了深入挖掘其中的原理，让我们看看<code>.git</code>目录下的文件都发生了哪些变化：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ GIT_DIR</span><span class="o">=</span><span class="nv">$PWD</span>/.git
</span><span class='line'><span class="nv">$ </span>cat <span class="nv">$GIT_DIR</span>/HEAD
</span><span class='line'>
</span><span class='line'>ref: refs/heads/master
</span><span class='line'>
</span><span class='line'><span class="nv">$ MY_CURRENT_BRANCH</span><span class="o">=</span><span class="k">$(</span>cat .git/HEAD | sed <span class="s1">&#39;s/ref: //g&#39;</span><span class="k">)</span>
</span><span class='line'><span class="nv">$ </span>cat <span class="nv">$GIT_DIR</span>/<span class="nv">$MY_CURRENT_BRANCH</span>
</span><span class='line'>
</span><span class='line'>cat: .git/refs/heads/master: No such file or directory
</span></code></pre></td></tr></table></div></figure>


<p>这里会返回一个错误信息，因为我们还没有在Git仓库中进行过任何提交，因此不会存在任何分支，包括默认分支<code>master</code>。</p>

<p>让我们进行一次提交，这时master分支会自动创建：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>git commit -m <span class="s2">&quot;Initial empty commit&quot;</span> --allow-empty
</span><span class='line'><span class="nv">$ </span>git branch
</span><span class='line'>
</span><span class='line'>* master
</span><span class='line'>
</span><span class='line'><span class="nv">$ </span>cat <span class="nv">$GIT_DIR</span>/<span class="nv">$MY_CURRENT_BRANCH</span>
</span><span class='line'>
</span><span class='line'>3bce5b130b17b7ce2f98d17b2998e32b1bc29d68
</span><span class='line'>
</span><span class='line'><span class="nv">$ </span>git cat-file -p <span class="k">$(</span>cat <span class="nv">$GIT_DIR</span>/<span class="nv">$MY_CURRENT_BRANCH</span><span class="k">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>输出的内容就是Git对象数据库中保存的信息了。</p>

<h2>6.6 对象数据库</h2>

<p><img src="http://www.aosabook.org/images/git/object-hierarchy.png" alt="图6.2：Git对象" /></p>

<p>图6.2：Git对象</p>

<p>Git有四种基本对象类型，版本库中的所有内容都是由这些基本对象类型构成的。每种对象类型包含以下属性：<em>类型</em>、<em>大小</em>、<em>内容</em>。这四种基本对象类型是：</p>

<ul>
<li><em>树</em>：用来表示目录结构，树中的元素可以是另一棵树或是一个二进制单元。</li>
<li><em>二进制单元</em>：表示一个文件。</li>
<li><em>提交</em>：提交会指向一个根节点树对象，并保存父提交的信息和其他基本信息。</li>
<li><em>里程碑</em>：里程碑有一个名称，并指向版本库中的一个提交对象。</li>
</ul>


<p>所有的基本对象类型使用SHA码来标识，它是一种40位的十六进制字符串，含有以下特性：</p>

<ul>
<li>如果两个对象是一致的，则SHA码一致。</li>
<li>如果两个对象不一致，则SHA码也不一致。</li>
<li>如果只是拷贝了对象的一部分，或者对象的数据发生了其他更改，只需重新计算其SHA码就能区别开来。</li>
</ul>


<p>前两种属性使得Git能够实施它的分布式模型（Git的第二个目标），第三种属性则是杜绝了数据混乱（第三个目标）。</p>

<h2>6.7 存储和压缩技术</h2>

<p>Git是通过压缩数据内容来解决存储大小的问题的，它使用一个索引文件来标识对象内容在压缩文件中的实际位置。</p>

<p><img src="http://www.aosabook.org/images/git/packed-format.png" alt="图6.3：压缩文件和它对应的索引文件" /></p>

<p>图6.3：压缩文件和它对应的索引文件</p>

<p>我们可以使用<code>git count-objects</code>来查看版本库中未经压缩的对象数量，然后让Git对未压缩的对象进行压缩，删除冗余的对象等。</p>

<p>Git对象的压缩方式进行过升级。过去，压缩文件和索引文件的CRC校验码会全部保存在索引文件中，这就无法检测出压缩对象中存在的数据混乱，因为后续压缩过程中不会再进行校验。新版本（Version 2）的压缩格式中将每个压缩对象的CRC校验码都保存了下来，从而解决了这一问题。同时，新版格式允许压缩文件大于4GB，这在以前是不支持的。为了更快地检测压缩文件是否损坏，文件末尾会保存一个20个字节的SHA1码，对压缩文件中所有对象的SHA码进行排序和校验。新版压缩格式的主要目的是为了满足Git设计目标中的杜绝数据混乱。</p>

<p>对于远程传输，Git会计算同步版本库（或分支）需要传输的提交和文件内容，生成相应的压缩文件，通过指定协议进行传输。</p>

<h2>6.8 记录合并历史</h2>

<p>上文我有提到过，Git和其它类RCS的版本控制系统的最大区别在于对合并历史的记录。如SVN将文件和目录结构的改动用线性提交来表示，版本号高的内容一定会覆盖版本号低的内容。因此，SVN不能直接提供分支功能，而是使用一种人为规定的目录结构来实现：</p>

<p><img src="http://www.aosabook.org/images/git/merge-history.png" alt="图6.4：合并历史的图形表示" /></p>

<p>图6.4：合并历史的图形表示</p>

<p>首先让我们用一个示例来说明要维护多个分支会多么麻烦，而且在某些场景下是有局限性的。</p>

<p>SVN中的“分支”通常放置在<code>branches/branch-name</code>，它和主干分支<code>trunk</code>（相当于 <em>master</em> ）目录同级。我们假设这个分支和主干分支是并行开发的。</p>

<p>举例来说，我们可能需要修改某个软件的数据库连接类型。在此过程中，我们想要将其他分支（非trunk）的内容合并到当前分支中。合并完成后（可能需要手工合并），我们继续修改。全部完成后，我们需要将<code>branches/branch-name</code>分支合并到<code>trunk</code>中。在类似SVN的线性历史版本控制系统中，我们无法得知其他分支的内容是否已经包含在trunk中了。</p>

<p>而对于以有向无环图为基础的版本控制系统（如Git）来说，就能很好地处理这种应用场景。如果某个分支不含没有合并至当前分支（如<code>db-migration</code>）的“提交”，我们就可以通过“提交”对象的继承关系来确定<code>db-migration</code>分支包含了那个分支的<code>HEAD</code>引用。由于“提交”对象可以包含零个或多个父提交，因此就能通过<code>db-migration</code>中的那次合并提交的信息来确定当前HEAD包含了两个分支中的内容。同理，当将<code>db-migration</code>合并至<code>master</code>分支时也能确认这些关系。</p>

<p>然而有一个问题无论是使用有向无环图还是线性提交都无法解决的，就是判断某个提交是否存在于每个分支中。例如上述例子中，我们假设已经将每个分支的提交都合并到各个分支去了。并不是所有情况下都是如此。</p>

<p>对于较为简单的情况，Git可以将其它分支的“提交”拣选（<code>cherry-pick</code>）到当前分支中，当然前提是这次提交必须是能够直接应用进来的。</p>

<h2>6.9 下一步做什么？</h2>

<p>上文提到，Git采用来自Unix世界的工具包设计理念，因此非常适合用来编写脚本。但是，当需要在长时间运行的应用程序或服务中内嵌Git工具库的话就不太容易了。虽然目前流行的IDE都提供了Git图形化界面，但开发这些工具所需花费的精力还是比其他版本控制系统要多，因为它们提供了便于使用的链接库。</p>

<p>为了解决这个问题，Shawn Pearce（来自谷歌开源程序办公室）率先实现了一个可供链接的Git类库，且发布协议较为宽松，因此没有阻碍该类库的推广。这个类库的名字是<a href="https://github.com/libgit2/libgit2">libgit2</a>。一开始它并不流行，直到一个名叫Vincent Marti的学生在谷歌编程夏令营中使用了它。从那以后，Vincent和Github持续对libgit2类库贡献代码，并为其他语言编写了相应类库，包括Ruby，Python，PHP，.NET，Lua，Object-C等。</p>

<p>Shawn Pearce还开启了一个名为<a href="https://github.com/eclipse/jgit">JGit</a>的BSD项目，使用纯Java语言实现，能够对Git版本库进行基本的操作。该类库现在由Eclipse基金会维护，用于Eclipse IDE的Git插件中。</p>

<p>还有其他一些有趣的周边项目，带有实验性质，使用各类数据源来保存Git对象，如：</p>

<ul>
<li><a href="https://github.com/spearce/jgit_cassandra">jgit_cassandra</a> 使用Apache Cassandra作为Git对象数据库。它是一种混合型的数据源，提供了动态的BigTable式的数据模型。</li>
<li><a href="https://github.com/spearce/jgit_hbase">jgit_hbase</a> 能够将Git对象保存在HBase中，一种KV型分布式数据库。</li>
<li><a href="https://github.com/libgit2/libgit2-backends">libgit2-backends</a> 由libgit2项目衍生而来，致力于提供其他种类的数据源，如Memcached，Redis，SQLite，MySQL。</li>
</ul>


<p>以上这些都是独立于Git核心工具包之外的项目。</p>

<p>如你所见，我们可以用各种方式来使用Git，它的表现形式不再只有命令行这一种了，而是成为一种版本控制系统的协议。</p>

<p>在本文撰写之时，这些项目都还没有发布稳定版本，所以还是有很多工作要做，但整体看来未来是光明的。</p>

<h2>6.10 经验教训</h2>

<p>在软件设计中，任何一个决定都有正反两面。作为一个在日常工作中大量使用Git，并且还为Git对象数据库开发了周边软件的程序员，我觉得Git目前的组织方式非常棒。因此，下文提到的“经验教训”更多的是来自其他开发者对于Git目前设计方式的不满，主要归咎于Git核心开发者当初做出的决定。</p>

<p>最常见的问题在于Git相较于其他CVS不能很好地和IDE进行整合，因为Git是基于工具包设计的，整合起来会比较具有挑战性。</p>

<p>早期Git的实现是采用shell脚本的方式，不能很好地跨平台，特别是对于Windows操作系统。虽然我相信Git开发者不会因为这个问题而寝食难安，但这的确阻碍了Git在大型公司内的推广。现在，有一个名为Git for Windows的项目由志愿者发起，及时地将最新的Git开发成果移植到Windows平台上。</p>

<p>Git工具包的设计方式所带来的另一个间接影响是，他的底层命令繁多，会让初学者陷入困境，难以理解Git出错时抛出的异常信息，最后无可适从。这就使得Git在某些开发团队中的推广受到阻碍。</p>

<p>即便如此，我仍然对Git核心项目以及其周边项目的开发充满信心。</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[使用Thrift API监控Storm集群]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/10/31/shi-yong-thrift-apijian-kong-stormji-qun/"/>
    <updated>2012-10-31T18:00:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/10/31/shi-yong-thrift-apijian-kong-stormji-qun</id>
    <content type="html"><![CDATA[<p><img class="right" src="http://www.gravatar.com/avatar/b881be65f3c4f45dd68bcf0fbe6ba82b.png"></p>

<p>Storm UI提供了基本的监控界面，可以查看当前时点集群内脚本的运行情况，其中比较重要的是消息吞吐量（Transferred）和处理延迟（Process latency）。不足的是，这套系统没有记录时序数据，因此想看一段时间内的趋势图，或是做脚本上下线的负载监控，Storm UI就无能为力了。</p>

<p>不过，Storm Nimbus开放了一套Thrift API，可以使用他获取各类信息。下面就介绍一下如何使用Python编写监控程序，定时获取脚本运行状态（吞吐量和延迟），并在监控系统中出图。</p>

<h3>安装Thrift API</h3>

<p><a href="http://thrift.apache.org/">Thrift</a>是Apache基金会下的一个项目，定义了一套面向服务程序的通信协议。它使用Thrift的<a href="http://thrift.apache.org/docs/idl/">IDL</a>定义接口，通过一个工具转换成不同语言的客户端，供其他程序调用。</p>

<p>可以<a href="http://archive.apache.org/dist/thrift/0.7.0/">点此下载Thrift源码包</a>，因为Storm使用的Thrift版本是0.7.0，所以建议安装该版本。</p>

<p>安装过程即 ./configure &amp;&amp; make &amp;&amp; sudo make install，安装完成后PATH会多出thrift这个命令。</p>

<h3>生成Storm Thrift API代码文件</h3>

<p>Storm Thrift API的定义文件在源码中提供，可以从Github中克隆相应的分支，并执行thrift命令：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ git clone -b 0.7.0 git://github.com/nathanmarz/storm.git
</span><span class='line'>$ cd storm/src
</span><span class='line'>$ thrift --gen py storm.thrift
</span><span class='line'>$ cd gen-py</span></code></pre></td></tr></table></div></figure>


<p>源码中似乎已经提供了py这个文件夹，也可以直接使用。</p>

<h3>获取某个Topology的吞吐量和延迟</h3>

<p>首先在Python项目中添加如下依赖：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>setup(install_requires=['thrift==0.7.0'])</span></code></pre></td></tr></table></div></figure>


<p>以下代码会找到集群中名为access_log的脚本，记录filter（Bolt）的吞吐量和延迟数。由于Storm Thrift API返回值的组织方式比较复杂，所以需要多多参考刚才生成的gen-py包中的内容。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="c">#连接到Storm Nimbus Thrift Server：</span>
</span><span class='line'>
</span><span class='line'><span class="n">socket</span> <span class="o">=</span> <span class="n">TSocket</span><span class="o">.</span><span class="n">TSocket</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">nimbus_host</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">nimbus_port</span><span class="p">)</span>
</span><span class='line'><span class="n">transport</span> <span class="o">=</span> <span class="n">TTransport</span><span class="o">.</span><span class="n">TFramedTransport</span><span class="p">(</span><span class="n">socket</span><span class="p">)</span>
</span><span class='line'><span class="n">protocol</span> <span class="o">=</span> <span class="n">TBinaryProtocol</span><span class="o">.</span><span class="n">TBinaryProtocol</span><span class="p">(</span><span class="n">transport</span><span class="p">)</span>
</span><span class='line'><span class="n">nimbus</span> <span class="o">=</span> <span class="n">Nimbus</span><span class="o">.</span><span class="n">Client</span><span class="p">(</span><span class="n">protocol</span><span class="p">)</span>
</span><span class='line'><span class="n">transport</span><span class="o">.</span><span class="n">open</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'><span class="c">#获取集群内脚本信息，得到脚本ID：</span>
</span><span class='line'>
</span><span class='line'><span class="n">cluster_info</span> <span class="o">=</span> <span class="n">nimbus</span><span class="o">.</span><span class="n">getClusterInfo</span><span class="p">()</span>
</span><span class='line'><span class="n">topology_id</span> <span class="o">=</span> <span class="bp">None</span>
</span><span class='line'><span class="k">for</span> <span class="n">topology</span> <span class="ow">in</span> <span class="n">cluster_info</span><span class="o">.</span><span class="n">topologies</span><span class="p">:</span>
</span><span class='line'><span class="k">if</span> <span class="n">topology</span><span class="o">.</span><span class="n">name</span> <span class="o">==</span> <span class="s">&#39;access_log&#39;</span><span class="p">:</span>
</span><span class='line'>    <span class="n">topology_id</span> <span class="o">=</span> <span class="n">topology</span><span class="o">.</span><span class="n">id</span>
</span><span class='line'>
</span><span class='line'><span class="c"># 获取脚本信息，计算吞吐量和延迟毫秒数</span>
</span><span class='line'>
</span><span class='line'><span class="n">topology_info</span> <span class="o">=</span> <span class="n">nimbus</span><span class="o">.</span><span class="n">getTopologyInfo</span><span class="p">(</span><span class="n">topology_id</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="n">transferred</span> <span class="o">=</span> <span class="mi">0</span>
</span><span class='line'><span class="n">filter_latencies</span> <span class="o">=</span> <span class="p">[]</span>
</span><span class='line'><span class="k">for</span> <span class="n">executor_info</span> <span class="ow">in</span> <span class="n">topology_info</span><span class="o">.</span><span class="n">executors</span><span class="p">:</span>
</span><span class='line'>    <span class="c"># 600表示10分钟内的均值</span>
</span><span class='line'>    <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">executor_info</span><span class="o">.</span><span class="n">stats</span><span class="o">.</span><span class="n">transferred</span><span class="p">[</span><span class="s">&#39;600&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">iteritems</span><span class="p">():</span>
</span><span class='line'>        <span class="n">transferred</span> <span class="o">+=</span> <span class="n">v</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">if</span> <span class="n">executor_info</span><span class="o">.</span><span class="n">component_id</span> <span class="o">==</span> <span class="s">&#39;filter&#39;</span><span class="p">:</span>
</span><span class='line'>        <span class="n">latencies</span> <span class="o">=</span> <span class="p">[]</span>
</span><span class='line'>        <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">executor_info</span><span class="o">.</span><span class="n">stats</span><span class="o">.</span><span class="n">specific</span><span class="o">.</span><span class="n">bolt</span><span class="o">.</span><span class="n">process_ms_avg</span><span class="p">[</span><span class="s">&#39;600&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">iteritems</span><span class="p">():</span>
</span><span class='line'>            <span class="n">latencies</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">v</span><span class="p">)</span>
</span><span class='line'>        <span class="n">filter_latencies</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">sum</span><span class="p">(</span><span class="n">latencies</span><span class="p">)</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">latencies</span><span class="p">))</span>
</span><span class='line'><span class="n">filter_latency</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">(</span><span class="n">filter_latencies</span><span class="p">)</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">filter_latencies</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>然后就可以将计算得到的transferred和filter_latency输出到监控系统中绘图了。</p>

<h3>注意事项</h3>

<p>在调试的过程中，有时会尝试用telnet查看Storm Thrift Server是否存活，一旦这样操作就会导致Nimbus挂起，无法自动退出。和作者确认过，这是Thrift自身的Bug，目前还没有解决方法。</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[深入Solr实战 - 实例篇]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/10/30/solr-guideline-in-action/"/>
    <updated>2012-10-30T10:29:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/10/30/solr-guideline-in-action</id>
    <content type="html"><![CDATA[<p><img class="right" src="http://0.gravatar.com/avatar/084c149bb5a1769082ef794c6dcd1a91"></p>

<p>之前我们有提到<a href="http://arch.corp.anjuke.com/blog/2012/10/15/solr-guildeline/">Solr的使用的一些规范</a>，这次我们深入实例。</p>

<p>我们从access log中找出一些查询耗时比较高的查询,跟去&#8221;深入solr实战&#8221;中提到的修改方式,做了一些更改,查询结果没有变化但是速度出现下降</p>

<!-- more -->


<p>测试环境:在cloud中不对外服务的solr instance,关闭filtercache，测试以下查询：</p>

<p>96ms左右</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>http://10.10.6.34:7731/solr/select/?
</span><span class='line'>start=0&
</span><span class='line'>q=commid:51087&
</span><span class='line'>qt=standard&
</span><span class='line'>fq=city_id:14&
</span><span class='line'>fq=hpstarttime:[0+TO+1347292898]&
</span><span class='line'>fq=hpendtime:[1347292898+TO++2147483647+]&
</span><span class='line'>fq=%28chpratio:3897.99871+AND+hpplanid:[0+TO+689944]%29+OR+one_chpratio:[3897.99872+TO+2147483647]&
</span><span class='line'>rows=0&
</span><span class='line'>version=2.2</span></code></pre></td></tr></table></div></figure>


<p>改成</p>

<p>40ms左右</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>http://10.10.6.34:7731/solr/select/?
</span><span class='line'>start=0&
</span><span class='line'>q=\*:\*&
</span><span class='line'>qt=standard&
</span><span class='line'>fq=commid:51087&
</span><span class='line'>fq=hpstarttime:[0+TO+1347292898]%20AND%20hpendtime:[1347292898+TO++2147483647+]%20AND%20%28%28chpratio:3897.99871+AND+hpplanid:[0+TO+689944]%29+OR+one_chpratio:[3897.99872+TO+2147483647+]%29&
</span><span class='line'>rows=0&
</span><span class='line'>version=2.2</span></code></pre></td></tr></table></div></figure>


<p>解析:</p>

<ol>
<li>这个instance存放的都是北京房源,所以拿掉了city_id:14的fq</li>
<li>hp的一系列fq首先不是term查询,其次相关的参数几乎时刻都在变化无法吃出cache,所以进行了合并</li>
<li>不需要计算score,所以让q=<em>:</em></li>
<li>commid相对固定,容易吃住cache,所以放在fq而没有合并到hp的查询条件中</li>
<li>hp的fq过滤后没有文档,所以commid是否合并如hp的fq对速度没有影响</li>
</ol>


<p>169ms左右</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>http://10.10.6.34:7731/solr/select/?
</span><span class='line'>fl=score,*&
</span><span class='line'>sort=order_string+desc,rank_level+asc,score+desc,rank_sub_level+asc,rank_score+desc&start=0&
</span><span class='line'>q=_val_:%22map%28rank_score2,4683.3,10000,0,1%29%22&
</span><span class='line'>qt=standard&
</span><span class='line'>hl=false&
</span><span class='line'>fq=city_id:14&
</span><span class='line'>fq=commid:50815&
</span><span class='line'>fq=islist:1&
</span><span class='line'>fq=hpendtime:[0+TO+1347292823]+OR+hpstarttime:[1347292823+TO+*]</span></code></pre></td></tr></table></div></figure>


<p>改成</p>

<p>35ms左右</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>http://10.10.6.34:7731/solr/select/?
</span><span class='line'>fl=score,*&
</span><span class='line'>sort=order_string+desc,rank_level+asc,score+desc,rank_sub_level+asc,rank_score+desc&
</span><span class='line'>start=0&
</span><span class='line'>q=_val_:%22map%28rank_score2,4683.3,10000,0,1%29%22&
</span><span class='line'>qt=standard&
</span><span class='line'>hl=false&
</span><span class='line'>fq=commid:50815%20AND%20islist:1%20AND%20hpendtime:[0+TO+1347292823]+OR+hpstarttime:[1347292823+TO+2147483647]</span></code></pre></td></tr></table></div></figure>


<p>解析</p>

<ol>
<li>同上1,去掉city_id:14的fq</li>
<li>同上2,合并fq</li>
<li>hp的fq过滤后有大量文档,所以合并commid可以明显提升速度</li>
</ol>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[LVM 简单实践]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/10/29/try-out-linux-lvm/"/>
    <updated>2012-10-29T21:20:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/10/29/try-out-linux-lvm</id>
    <content type="html"><![CDATA[<p><img class="right" src="http://0.gravatar.com/avatar/ce1e13bbf946c92e2abf740f8909bafa"></p>

<p>为什么会想到要使用lvm呢，因为家里的主力下载机是一台PC，她又是我的主要平时使用的机子（除开mac外），经常要对下载那个分区进行整理，装个游戏，压制点视频音乐有时候分区会很紧张，然而跨分区是很复杂的操作。这就让我想到了linux下面的lvm功能。</p>

<p><img class="center" src="http://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/LVM1.svg/500px-LVM1.svg.png"></p>

<!-- more -->


<h2>什么是lvm</h2>

<p><a href="http://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)">lvm</a>是Linux核心提供的逻辑卷轴管理功能，主要应用在磁盘分区上；传统的计算机使用磁盘主要通过管理分区，在常见的MBR磁盘上最多只可以使用4个主分区，或者任意个逻辑分区；虽然在GPT分区上可以实现使用任意分区数量，但是分区大小总是有限制的，即使很多文件系统工具提供了对分区大小进行热操作，仍旧不是很方便。lvm则可以通过让系统核心支持真正的逻辑卷管理，达到物理分区的虚拟化。可以随时对逻辑卷进行大小的修改。</p>

<h2>使用</h2>

<p>更加系统的讲解请参考<a href="http://tldp.org/HOWTO/LVM-HOWTO/">LVM-HOWTO</a>。</p>

<p>注意我们使用的lvm2版本，主要区别，可以查看<a href="http://tldp.org/HOWTO/LVM-HOWTO/lvm2faq.html">这里</a>。</p>

<p>我下面的演示，主要基于gentoo，安装的时候livecd里已经带了lvm工具。</p>

<p>解释一些lvm里使用的名词，先给一张图</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>+-- Volume Group --------------------------------+
</span><span class='line'>|                                                |
</span><span class='line'>|    +----------------------------------------+    |
</span><span class='line'>| PV | PE |  PE | PE | PE | PE | PE | PE | PE |    |
</span><span class='line'>|    +----------------------------------------+    |
</span><span class='line'>|      .            .          .        .          |
</span><span class='line'>|      .          .            .        .          |
</span><span class='line'>|    +----------------------------------------+    |
</span><span class='line'>| LV | LE |  LE | LE | LE | LE | LE | LE | LE |    |
</span><span class='line'>|    +----------------------------------------+    |
</span><span class='line'>|            .          .        .           .     |
</span><span class='line'>|            .            .        .         .     |
</span><span class='line'>|    +----------------------------------------+    |
</span><span class='line'>| PV | PE |  PE | PE | PE | PE | PE | PE | PE |    |
</span><span class='line'>|    +----------------------------------------+    |
</span><span class='line'>|                                                |
</span><span class='line'>+------------------------------------------------+</span></code></pre></td></tr></table></div></figure>


<ul>
<li>volume group (VG): lvm中使用的最高层次的抽象对象，它把(物理上的)逻辑和主分区划为管理单元</li>
<li>physical volume (PV): 一个pv可以是一块磁盘，或者是看起来磁盘设备 (例如/dev/sda1)</li>
<li>logical volume (LV): 和非lvm系统中的逻辑分区一样，一个lv看起来就像是普通的块设备，而它可以包含文件系统(例如/home)</li>
<li>physical extend (PE): 每个pv都被分为几部分数据，它们就是pe，对于lvm来说，它们和le大小一样。</li>
<li>logical extend (LE): 类似PE</li>
</ul>


<p>最后，更加直观地理解，可以看下图，</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>
</span><span class='line'>    hda1   hdc1      (PV:s on partitions or whole disks)
</span><span class='line'>       \   /
</span><span class='line'>        \ /
</span><span class='line'>       diskvg        (VG)
</span><span class='line'>       /  |  \
</span><span class='line'>      /   |   \
</span><span class='line'>  usrlv rootlv varlv (LV:s)
</span><span class='line'>    |      |     |
</span><span class='line'> ext2  reiserfs  xfs (filesystems)
</span></code></pre></td></tr></table></div></figure>


<h2>要求</h2>

<ol>
<li>使用lvm，至少需要2.4.x的kernel</li>
<li>安装了lvm</li>
</ol>


<h2>配置</h2>

<h3>背景</h3>

<p>这里解释的是在gentoo安装过程中将两块磁盘中的分区配置成使用lvm的过程。</p>

<p>主要参考了<a href="http://www.gentoo.org/doc/en/lvm2.xml">gentoo的lvm手册</a>。</p>

<p>假设，我现在有两块SATA硬盘，将其物理分区安排如下</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>/dev/sda1  -&gt; /boot
</span><span class='line'>/dev/sda2  -&gt; (swap)
</span><span class='line'>/dev/sda3  -&gt; /
</span><span class='line'>/dev/sda4  -&gt; 将使用lvm
</span><span class='line'>/dev/sdb1  -&gt; 将使用lvm</span></code></pre></td></tr></table></div></figure>


<p><strong>请注意操作的安全性，一个错误的操作可能会导致整个分区的数据丢失!!</strong></p>

<h3>安装</h3>

<ul>
<li>使用fdisk或者你喜欢的分区工具分区</li>
<li>启动lvm服务，在gentoo下使用<code>/etc/init.d/lvm start</code>，livecd里默认已经启动</li>
<li>配置lvm，打开<code>/etc/lvm/lvm.conf</code>，修改如下信息</li>
</ul>


<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class=''><span class='line'># a表示add添加，r表示reject忽略
</span><span class='line'># 语法类似正则，表示把/dev/sda与/dev/sdb里的分区都加入到lvm的识别里
</span><span class='line'>filter = [ "a|/dev/sd[ab]|", "r/.*/" ]
</span><span class='line'>
</span><span class='line'># 使之前配置的vg都生效
</span><span class='line'>vgscan
</span><span class='line'>vgchange -a y</span></code></pre></td></tr></table></div></figure>


<h1>把分区准备好</h1>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>pvcreate /dev/sda4 /dev/sdb1</span></code></pre></td></tr></table></div></figure>


<p>我们这个例子里，<code>/dev/sda1</code>，<code>/dev/sda2</code>，<code>/dev/sda3</code>分别是<code>/boot</code>,<code>swap</code>和<code>/</code></p>

<p><strong>极其不推荐在/etc, /lib, /mnt, /proc, /sbin, /dev, and /root上使用lvm，可能会导致我些目录无法访问!!</strong></p>

<ul>
<li>创建并扩展一个vg</li>
</ul>


<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>vgcreate vg /dev/sad4
</span><span class='line'>vgextend vg /dev/sdb1</span></code></pre></td></tr></table></div></figure>


<ul>
<li>创建最终要使用的lv</li>
</ul>


<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>lvcreate -L10G -nusr vg # 在vg里创建一个大小为10G的lv，名字是usr
</span><span class='line'>lvcreate -L5G -nhome vg # 在vg里创建一个大小为5G的lv，名字是home
</span><span class='line'>lvcreate -L10G -nvar vg # 在vg里创建一个大小为10G的lv，名字是var
</span><span class='line'>
</span><span class='line'>lvextend -L+5G /dev/vg/home # 举例，给home这个lv增加5G容量</span></code></pre></td></tr></table></div></figure>


<ul>
<li>在lv上创建文件系统，并挂载</li>
</ul>


<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>mkfs.reiserfs /dev/vg/usr
</span><span class='line'>mkfs.reiserfs /dev/vg/home
</span><span class='line'>mkfs.reiserfs /dev/vg/var
</span><span class='line'>
</span><span class='line'># 在这之前先挂载必要分区，请参考gentoo安装手册
</span><span class='line'>mkdir /mnt/gentoo/usr
</span><span class='line'>mount /dev/vg/usr /mnt/gentoo/usr
</span><span class='line'>mkdir /mnt/gentoo/home
</span><span class='line'>mount /dev/vg/usr /mnt/gentoo/home
</span><span class='line'>mkdir /mnt/gentoo/var
</span><span class='line'>mount /dev/vg/usr /mnt/gentoo/var</span></code></pre></td></tr></table></div></figure>


<ul>
<li>下面继续gentoo的安装过程，chroot一直到配置kernel</li>
<li>配置kernel，打开lvm的支持选项</li>
<li>2.4.x的kernel</li>
</ul>


<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>Multi-device support (RAID and LVM)  ---&gt;
</span><span class='line'>  [*] Multiple devices driver support (RAID and LVM)
</span><span class='line'>  &lt; &gt;  RAID support
</span><span class='line'>(注意这里不选择LVM是有意的，它的意思是支持LVM1)
</span><span class='line'>  &lt; &gt;  Logical volume manager (LVM) support
</span><span class='line'>  &lt;M&gt;  Device-mapper support
</span><span class='line'>  &lt; &gt;   Mirror (RAID-1) support</span></code></pre></td></tr></table></div></figure>


<ul>
<li>2.6.x以及之后版本的kernel</li>
</ul>


<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>Device Drivers  ---&gt;
</span><span class='line'> Multiple devices driver support (RAID and LVM) ---&gt;
</span><span class='line'>   [*] Multiple devices driver support (RAID and LVM)
</span><span class='line'>   &lt; &gt;   RAID support
</span><span class='line'>   &lt;M&gt;   Device mapper support</span></code></pre></td></tr></table></div></figure>


<p>编译完成后的模块名字叫<code>dm-mod.ko</code></p>

<p><strong>请注意，务必让<code>/usr/src/linux</code>指向您正在使用的kernel核心源代码，因为lvm的ebuild会根据路径查找device-mapper的依赖性</strong></p>

<ul>
<li><p>配置chroot里的lvm.conf，和外层一致</p></li>
<li><p>配置<code>/etc/fstab</code>，根据需要配置lvm的分区</p></li>
</ul>


<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>/dev/sda1     /boot        ext2       defaults      1 2
</span><span class='line'>/dev/sda2     none         swap       sw            0 0
</span><span class='line'>/dev/sda3     /            reiserfs   noatime       0 1
</span><span class='line'>/dev/cdrom    /mnt/cdrom   auto       noauto,ro     0 0
</span><span class='line'># Logical Volumes
</span><span class='line'>/dev/vg/usr   /usr         reiserfs   noatime       0 2
</span><span class='line'>/dev/vg/home  /home        reiserfs   noatime       0 2
</span><span class='line'>/dev/vg/var   /var         reiserfs   noatime       0 2</span></code></pre></td></tr></table></div></figure>


<ul>
<li>继续其他安装配置，一直到退出chroot前</li>
<li>安装lvm，并将其设置为启动</li>
</ul>


<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>emerge -v lvm2
</span><span class='line'>rc-update add lvm boot</span></code></pre></td></tr></table></div></figure>


<ul>
<li>退出chroot，重启前，关闭所有lvm分区</li>
</ul>


<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>vgchange -a n</span></code></pre></td></tr></table></div></figure>


<h3>如果中途有重启</h3>

<p>如果在安装过程中因为我些原因需要重启，需要按以下操作重新激活lvm分区</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>vgscan --mknodes</span></code></pre></td></tr></table></div></figure>


<p>而一些比较旧的安装CD里需要按照以下顺序激活</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class=''><span class='line'># 先关闭所有vg
</span><span class='line'>vgchange -a n
</span><span class='line'># 导出所有vg
</span><span class='line'>vgexport -a
</span><span class='line'># 导入所有vg
</span><span class='line'>vgimport -a
</span><span class='line'># 重新激活所有vg
</span><span class='line'>vgchange -a y</span></code></pre></td></tr></table></div></figure>


<h3>使用</h3>

<p>如果没有意外，那么重启后系统可以成功boot进入新安装的gentoo系统</p>

<p>我的gentoo box的磁盘情况如下</p>

<p><img src="http://arch.corp.anjuke.com/medias/20121029/lvm.jpeg" alt="lvm" /></p>

<p>其实也是这个我才知道前阵子上线的一台服务器上<code>/dev/mapper</code>，原来其实就是lvm啊XDDD</p>

<p>__END__</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[深入Solr实战]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/10/15/solr-guildeline/"/>
    <updated>2012-10-15T15:03:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/10/15/solr-guildeline</id>
    <content type="html"><![CDATA[<p><img class="right" src="http://0.gravatar.com/avatar/084c149bb5a1769082ef794c6dcd1a91"></p>

<p>solr为我们提供便捷的全文检索服务,查询标准,更新容易,部署方便.我们来介绍一下如何更好的使用solr来为我们服务</p>

<!-- more -->


<h1>新建solr服务</h1>

<h3>新建solr服务时,需要大家提供以下信息:</h3>

<ol>
<li><p>servicename</p></li>
<li><p>预计查询、更新频率</p></li>
<li><p>多少文档数，每个文档有多少field,field的类型，有多少field是store的</p></li>
<li><p>常用查询语句</p></li>
</ol>


<p>servicename是提供solr服务的url的一部分</p>

<p>预计查询频率是指访问高峰期solr每分钟的请求次数</p>

<p>更新频率是指全天solr更新最频繁的时段,每分钟会提交多少个文档更新</p>

<p>每个文档有多少个field,每个field的类型,是否是store的,可以预估solr服务的磁盘及内存消耗</p>

<p>提供常用的查询语句,可以纠正一些不正确的写法,预估cpu的消耗</p>

<h3>一些建议:</h3>

<p>所有业务字段除去id以外都不要store</p>

<p>为了调试和排查错误方便,可以有一个updatetime的store字段,用于记录当前solr数据被update的时间</p>

<h4>旁白:</h4>

<p>关于字段store的测试,我们使用prop00服务,当时有245w的文档.我们将所有字段,除ID外进行了store和非store的调整,得到了下面的测试结果.</p>

<table>
<thead>
<tr>
<th></th>
<th>非store</th>
<th>store</th>
</tr>
</thead>
<tbody>
<tr>
<td>索引文件大小</td>
<td>767m</td>
<td>1552m</td>
</tr>
<tr>
<td>VSZ</td>
<td>2527m</td>
<td>2608m</td>
</tr>
<tr>
<td>RSS</td>
<td>2185m</td>
<td>2194m</td>
</tr>
<tr>
<td>10-11点90%时间</td>
<td>119ms</td>
<td>179ms</td>
</tr>
</tbody>
</table>


<p><strong>想速度快一点吗?非store吧</strong></p>

<h1>使用solr服务</h1>

<p>solr服务对于应用来说,就是更新和查询.而更新大家都知道通过包含unique的id的xml文件进行单个或者批量更新,这里说得更多的是如何写一个快一点的查询</p>

<h2>理论知识(可以跳过)</h2>

<h3>1. start</h3>

<p>和数据库一样，使用大的start会对solr造成很大的压力，因此，start一定要限制在很小的范围内。</p>

<p>如 ajk-qa 有以下查询：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='text'><span class='line'>fl=id&amp;sort=quetime+asc&amp;start=557550&amp;q=*:*&amp;wt=json&amp;rows=50
</span></code></pre></td></tr></table></div></figure>


<p>这句话查询时间要1.5s左右。</p>

<p><u><strong>结论:类似db的游标翻页,通过quetime>某个值来过滤，减少start的值。</strong></u></p>

<h3>2. 关键词查询时间复杂度</h3>

<h4>2.1 单个term查询:</h4>

<p>假设 n为命中的doc数量，k为寻找前面k条记录，对应solr的start+rows</p>

<p>最糟糕的情形下，时间复杂度为 n * log(k) 其中log(k)主要为调整最小堆所花费的时间。</p>

<p>最好的情形下：为n，命中的doc的score都一样，无需调整堆。</p>

<p><u><strong>结论：单个term性能主要决定于k，若k很大，会对性能有较多影响。</strong></u></p>

<h4>2.2 多个term or查询：</h4>

<p>假设 n为单个term平均命中doc数量，m为term个数，k为寻找前面k条记录</p>

<p>最糟糕的情形下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='text'><span class='line'>n*m + n*m*log(m) + n*m + n * m * log(k) = n * m (log(m) + log(k))
</span></code></pre></td></tr></table></div></figure>


<p>较前面的单个term查询，多出来log(m)，其为调整各个term对应的scorer最小堆所需的时间。</p>

<p>最好的情形下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='text'><span class='line'>n*m， 一样的道理，两个堆都无需调整。
</span></code></pre></td></tr></table></div></figure>


<p><u><strong>结论：多个term or时多了log(m)，因此term的数量不宜过大。</strong></u></p>

<h4>2.3 多个term and查询：</h4>

<p>假设 n为单个term平均命中doc数量，ni为第i个term命中的文档数，m为term个数，N 为最终命中的doc数量，k为寻找前面k条记录</p>

<p>最糟糕的情形下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='text'><span class='line'>N*log(k) + max(ni)
</span></code></pre></td></tr></table></div></figure>


<p>最好的情形下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='text'><span class='line'>N + min(ni)
</span></code></pre></td></tr></table></div></figure>


<p>找到N条记录后，选择k条最前面的记录时，堆也不用调整， 各个term的对应的doc都差不多分布集中在一起，即</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='text'><span class='line'>min(docend-docbegin) = n
</span></code></pre></td></tr></table></div></figure>


<p><u><strong>结论：多个term and时和单个term基本一样。</strong></u></p>

<h3>3.facet query</h3>

<h4>3.1 facet.field: n 为命中的doc数量</h4>

<p>基本流程是：solr里面遍历q命中的n个文档，然后统计出各个field value对的数量,</p>

<h4>3.2 facet.query: 同搜索，取决于facet的查询，将facet.query查出来的文档集合和q查出来的文档集合做交集，然后统计个数。</h4>

<p>注：由于这个facet.field需要从索引中获得field对应的term信息，因此使用facet.field并不一定比facet.query快。</p>

<p><u><strong>结论：facet query和q一样需要注意性能问题。</strong></u></p>

<h3>4.function boost</h3>

<p>和一般查询一样，只是修改了scorer，没有特别增加的时间复杂度</p>

<p><u><strong>结论：不是复杂的function boost对性能影响不大。</strong></u></p>

<h3>5.fq的查询过程</h3>

<h4>5.1 查询过程</h4>

<p>solr这边fq会先从索引中查出对应的文档集合,每个fq都有对应的filtercache，然后做交集,然后从这些文档中去检索满足q条件的doc.</p>

<h4>5.2 fq 特点：</h4>

<p>fq有filtercache。q没有cache</p>

<p>对最终的结果来说，fq不计算分数。q会计算</p>

<p>fq在单个term查询时效率优于q，因为不需要计算分数, solr直接调用lucene的接口，获得倒排表命中的文档.fq在除去单个term查询的情况以外, 则调用lucene的search接口，效率都和q一样</p>

<p>多个非term查询会导致solr调用lucene的多次search，会非常慢，应该合并。</p>

<h4>5.3 多个fq查询和fq里面多个查询AND的区别如下：</h4>

<p>多个fq 进行 and时，使用的lucene的OpenBitSet(java中的BitSet)对fq结果集做并集.</p>

<p>多个fq是每个fq查出来的结果后进行交集</p>

<p>fq里面多个查询是一边查询一边做交集，如果fq有一个查询对应的文档数很少，建议使用fq多个查询AND</p>

<h4>5.4 使用fq时的建议：</h4>

<p>如果fq里面有非term查询，建议将非term查询合并在一起</p>

<p>如果是仅仅根据单个term过滤结果，又不需要排序，建议使用<code>fq，q=*:*</code></p>

<p>合理精简条件，solr不会对fq优化，去除一个fq可以少一次查询. fq如果并不能起到过滤文档的作用,请不要放在url中.例如:查询ajk-prop11的url中fq=city_id:11就可以拿掉</p>

<p>如果查询中有出现非term查询,可以将非term查询合并到当前查询中命中文档数最少的fq中</p>

<h3>6. q里面的and查询过程</h3>

<h4>6.1查询时间复杂度的计算</h4>

<p>lucene在score nextdoc的时候，就对根据term找到的doc做并集处理，该项动作的时间复杂度最好为O(N) N为满足所有条件的文档数，其实就是 docend-docbegin</p>

<p>选择k个文档的时间复杂度是一样的： <code>N * log(k)</code>， <code>N</code>为满足所有条件的文档数</p>

<h4>6.2 使用q时的建议:</h4>

<p>功能上，fq没有score的功能，而q里面可以score。</p>

<p>效率上，fq 和 q查询的差异在于找到满足条件文档的时间复杂度，一个是<code>O(min(n)) * m + O(Nf)</code>，一个是<code>O(N)</code>，如果fq的结果是经常变的，建议使用q里面的and条件。</p>

<h3>7. 关于dismax的mm</h3>

<p><code>mm=2</code>表示q里面的字句任意两个匹配就返回,造成大量的term查询</p>

<h3>8. 范围查询</h3>

<p>范围查询<code>fq=hpstarttime:[* TO 1351354156] OR hpendtime:[1351354156 TO *]</code>,范围越大越耗时,这种查询对速度影响非常大,且从查询语句上没有任何办法优化.</p>

<p>lucene存的都是针对term的倒排表,即不是数值类型而是多个字符串，不像数据库那样会有某个字段的index.</p>

<p>schema中可以调整<code>precisionStep=4</code>的值,默认是4或者没有设置,这个值越大则索引小性能差，越小则索引大性能理论上的好,精度高(目前还未测试)</p>

<p>建议:</p>

<blockquote><p>缩小范围，这是最直接最管用的方法</p>

<p>不要使用<code>*</code>而是根据业务使用一个具体的数值,比如当前时间</p></blockquote>

<h4>9. 其他建议</h4>

<p>solr的机器的内存使用率不能超过80%</p>

<p>optimize虽然官方声明是deprecated的,但是实际用下来还是需要继续</p>

<p>通过<code>solr/admin/luke?fl=normalanswerstr&amp;numTerms=100</code>分析term,过滤掉没有意义的term</p>

<p>附：什么是单个term查询：就是指定field 以及 value，且没有二元逻辑运算符，如 <code>city_id:14</code>， 注意：<code>field:(val1 OR val2)</code> 不是单个term查询。</p>

<h2>结论(精华)</h2>

<p><u><strong>结论:</strong></u> start+rows过大会严重影响solr的查询速度.</p>

<p>解决方案:类似db的游标翻页,通过加入其他过滤条件(如<code>id&gt;某个值</code>)<code>quetime&gt;</code>，减少start的值.</p>

<p>建议:start值<code>+rows</code>的值不要超过2000</p>

<hr />

<p><u><strong>结论:</strong></u> term数量在or查询中严重影响solr的查询数度</p>

<p>解决方案:减少term的个数</p>

<p>建议:term数量控制不超过10个</p>

<hr />

<p><u><strong>结论</strong></u>:每个fq都会有filtercache,对于更新太频繁的solr没有效果</p>

<p>解决方案:减少update频率,减少fq的查询条件数目</p>

<p>建议:daily build,减少自定义条件查询,或者将fq放到q的and条件中</p>

<hr />

<p><u><strong>结论:</strong></u> q可以计算分数,而fq不会,对于不需要score排序的查询,可以不使用q</p>

<p>解决方案:非score排序,可以使用q=<em>:</em>&amp;fq=field1:value1而不是q=field1:value1</p>

<hr />

<p><u><strong>结论:</strong></u>多个fq查询,先对每个fq进行查询再做交集.而在fq里做多个and查询,会先优化查询顺序然后再边查询边交集查询</p>

<p>建议:如果fq中有一个可以快速过滤文档的查询,可以将其他fq查询合并入此fq中</p>

<hr />

<p><u><strong>结论:</strong></u> 每次非term查询,会使solr调用lucene的search</p>

<p>建议:如果有多个fq进行非term查询,建议合并(但是合并会导致fq的filtercache命中率低,需要平衡,另外可以结合上面fq and查询结论,合并到一个可以快速过滤文档的fq查询中)</p>

<hr />

<p><u><strong>结论:</strong></u> solr不会对多个fq进行优化,只会对单个fq内部优化</p>

<p>建议:对不过滤文档的fq可以去除</p>

<hr />

<p><u><strong>结论:</strong></u> dismax的mm参数,是从q中取出制定数据的关键字进行组合然后进行or查询,会造成大量的term查询</p>

<p>解决方案:<code>mm=1%</code>和<code>mm=99%</code>是一样的情况,只有拿掉mm使成为and查询</p>

<hr />

<p><u><strong>结论:</strong></u> solr的数值在存储是依然使用的是若干字符串进行存储,所以范围查询中范围越大越耗时</p>

<p>解决方案:缩小查询范围</p>

<p>建议:不要使用<code>*</code>,而是用一个根据业务使用一个具体的值</p>

<hr />

<p>参考资料</p>

<ul>
<li><p><a href="http://stackoverflow.com/questions/6462350/is-filtering-faster-than-querying-in-lucene">http://stackoverflow.com/questions/6462350/is-filtering-faster-than-querying-in-lucene</a></p></li>
<li><p><a href="http://xangqun.iteye.com/blog/686840">http://xangqun.iteye.com/blog/686840</a></p></li>
<li><p><a href="http://www.wanghd.com/blog/2012/09/21/solrde-shu-zhi-cun-chu-he-fan-wei-cha-xun/">http://www.wanghd.com/blog/2012/09/21/solrde-shu-zhi-cun-chu-he-fan-wei-cha-xun/</a></p></li>
<li><p><a href="http://hadoopcn.iteye.com/blog/1550402">http://hadoopcn.iteye.com/blog/1550402</a></p></li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Mogilefs设备权重算法]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/10/09/how-does-mogilefs-select-device-for-upload/"/>
    <updated>2012-10-09T17:55:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/10/09/how-does-mogilefs-select-device-for-upload</id>
    <content type="html"><![CDATA[<p><img class="right" src="http://0.gravatar.com/avatar/78811efe9c142e0898c049686e25594d"></p>

<h3>遇到的问题</h3>

<ul>
<li>有几块设备已经剩余10%容量，有几块设备剩余75%容量，为什么75%的util相当高呢</li>
<li>原本理解mogilefs平均分配新文件到设备上，猜想应该是不对的</li>
</ul>


<h3>看看mogilefs到底是怎么做的呢</h3>

<ul>
<li>处理文件是Query.pl干的，确认新建文件是cmd_create_open函数所为</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class='perl'><span class='line'><span class="k">sub </span><span class="nf">cmd_create_open</span> <span class="p">{</span>
</span><span class='line'>    <span class="c1">#略</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">#默认情况下设备的选择是从这开始的，参数是所有设备</span>
</span><span class='line'>    <span class="nv">@devices</span> <span class="o">=</span> <span class="n">sort_devs_by_freespace</span><span class="p">(</span><span class="nn">Mgd::</span><span class="n">device_factory</span><span class="p">()</span><span class="o">-&gt;</span><span class="n">get_all</span><span class="p">);</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">#略</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<ul>
<li>sort_devs_by_freespace用剩余百分比做为权重，生成一个二维数组做参数[前20个]</li>
<li>真正的返回值是通过MogileFS::Util::weighted_list进行权重计算</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
</pre></td><td class='code'><pre><code class='perl'><span class='line'><span class="k">sub </span><span class="nf">sort_devs_by_freespace</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">my</span> <span class="nv">@devices_with_weights</span> <span class="o">=</span> <span class="nb">map</span> <span class="p">{</span> <span class="c1">#生成二维数组，剩余大小用来做权重</span>
</span><span class='line'>        <span class="p">[</span><span class="nv">$_</span><span class="p">,</span> <span class="mi">100</span> <span class="o">*</span> <span class="nv">$_</span><span class="o">-&gt;</span><span class="n">percent_free</span><span class="p">]</span>
</span><span class='line'>    <span class="p">}</span> <span class="nb">sort</span> <span class="p">{</span>  <span class="c1">#排序，从大到小排序</span>
</span><span class='line'>        <span class="nv">$b</span><span class="o">-&gt;</span><span class="n">percent_free</span> <span class="sr">&lt;=&gt;</span> <span class="nv">$a</span><span class="o">-&gt;</span><span class="n">percent_free</span><span class="p">;</span>
</span><span class='line'>    <span class="p">}</span> <span class="nb">grep</span> <span class="p">{</span>  <span class="c1">#过滤设备，只要可以新建文件的，就是状态为alive的</span>
</span><span class='line'>        <span class="nv">$_</span><span class="o">-&gt;</span><span class="n">should_get_new_files</span><span class="p">;</span>
</span><span class='line'>    <span class="p">}</span> <span class="nv">@_</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">my</span> <span class="nv">@list</span> <span class="o">=</span>
</span><span class='line'>        <span class="nn">MogileFS::Util::</span><span class="n">weighted_list</span><span class="p">(</span><span class="nb">splice</span><span class="p">(</span><span class="nv">@devices_with_weights</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">20</span><span class="p">));</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">return</span> <span class="nv">@list</span><span class="p">;</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<ul>
<li>这里才是算法的核心，让容量多的多放点文件，但当容量很大时基本随即，做个例子会比较直观</li>
<li>假设有两个设备，剩余的容量分别是10和75</li>
<li>那么sum就是85，如果乘以随机数的话可能是8.5(0.1)或者是76.5(0.9)</li>
<li>如果是8.5，那么75就大于8.5了，先返回了【并且这种概率大】</li>
<li>如果是76.5，那么75就不大于76.5了, 则返回另一个【这种概率小】</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
</pre></td><td class='code'><pre><code class='perl'><span class='line'><span class="k">sub </span><span class="nf">weighted_list</span> <span class="p">(@)</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">my</span> <span class="nv">@list</span> <span class="o">=</span> <span class="nb">grep</span> <span class="p">{</span> <span class="nv">$_</span><span class="o">-&gt;</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">&gt;</span> <span class="mi">0</span> <span class="p">}</span> <span class="nv">@_</span><span class="p">;</span> <span class="c1">#排除掉等于0的</span>
</span><span class='line'>    <span class="k">my</span> <span class="nv">@ret</span><span class="p">;</span> <span class="c1">#初始化返回数组</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">my</span> <span class="nv">$sum</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</span><span class='line'>    <span class="nv">$sum</span> <span class="o">+=</span> <span class="nv">$_</span><span class="o">-&gt;</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="k">foreach</span> <span class="nv">@list</span><span class="p">;</span> <span class="c1">#累加剩余容量</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">my</span> <span class="nv">$getone</span> <span class="o">=</span> <span class="k">sub </span><span class="p">{</span>
</span><span class='line'>        <span class="k">return</span> <span class="nb">shift</span><span class="p">(</span><span class="nv">@list</span><span class="p">)</span><span class="o">-&gt;</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</span><span class='line'>            <span class="k">if</span> <span class="nb">scalar</span><span class="p">(</span><span class="nv">@list</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">;</span> <span class="c1">#只有一个了，直接返回</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">my</span> <span class="nv">$val</span> <span class="o">=</span> <span class="nb">rand</span><span class="p">()</span> <span class="o">*</span> <span class="nv">$sum</span><span class="p">;</span> <span class="c1">#总容量乘以0到1的随机数</span>
</span><span class='line'>        <span class="k">my</span> <span class="nv">$curval</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</span><span class='line'>        <span class="k">for</span> <span class="p">(</span><span class="k">my</span> <span class="nv">$idx</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="nv">$idx</span> <span class="o">&lt;</span> <span class="nb">scalar</span><span class="p">(</span><span class="nv">@list</span><span class="p">);</span> <span class="nv">$idx</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>            <span class="k">my</span> <span class="nv">$item</span> <span class="o">=</span> <span class="nv">$list</span><span class="p">[</span><span class="nv">$idx</span><span class="p">];</span>
</span><span class='line'>            <span class="nv">$curval</span> <span class="o">+=</span> <span class="nv">$item</span><span class="o">-&gt;</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span> <span class="c1">#累加进去,保证肯定有返回</span>
</span><span class='line'>            <span class="k">if</span> <span class="p">(</span><span class="nv">$curval</span> <span class="o">&gt;=</span> <span class="nv">$val</span><span class="p">)</span> <span class="p">{</span> <span class="c1">#如果curval比随机乘积大</span>
</span><span class='line'>                <span class="k">my</span> <span class="p">(</span><span class="nv">$ret</span><span class="p">)</span> <span class="o">=</span> <span class="nb">splice</span><span class="p">(</span><span class="nv">@list</span><span class="p">,</span> <span class="nv">$idx</span><span class="p">,</span> <span class="mi">1</span><span class="p">);</span> <span class="c1">#切割出来</span>
</span><span class='line'>                <span class="nv">$sum</span> <span class="o">-=</span> <span class="nv">$item</span><span class="o">-&gt;</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span> <span class="c1">#从总容量里减去</span>
</span><span class='line'>                <span class="k">return</span> <span class="nv">$ret</span><span class="o">-&gt;</span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>   <span class="c1">#返回这个设备</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>    <span class="p">};</span>
</span><span class='line'>
</span><span class='line'>    <span class="nb">push</span> <span class="nv">@ret</span><span class="p">,</span> <span class="nv">$getone</span><span class="o">-&gt;</span><span class="p">()</span> <span class="k">while</span> <span class="nv">@list</span><span class="p">;</span> <span class="c1">#循环执行getone，加到返回数组</span>
</span><span class='line'>    <span class="k">return</span> <span class="nv">@ret</span><span class="p">;</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>



]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[reviewboard 简要使用介绍]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/10/09/reviewboard-howto/"/>
    <updated>2012-10-09T09:06:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/10/09/reviewboard-howto</id>
    <content type="html"><![CDATA[<p><img class="right" src="http://0.gravatar.com/avatar/ce1e13bbf946c92e2abf740f8909bafa"></p>

<h2>Hold a sec! What is reviewboard ?</h2>

<p><a href="http://www.reviewboard.org/">reviewboard</a>是一款基于web的强大的code
review工具，他提供了便捷、强大的功能来帮助开发者更顺利地进行code review这一核心工作。支持众多版本控制系统。</p>

<!-- more -->


<h2>reviewboard in Anjuke Inc.</h2>

<p>在安居客，我们使用reviewboard来进行code review，他的地址是</p>

<p><a href="http://git.corp.anjuke.com/reviews">http://git.corp.anjuke.com/reviews</a></p>

<p>使用域帐号登录</p>

<p><img class="center" src="http://arch.corp.anjuke.com/images/2012/10/reviewboard001.jpg"></p>

<p>登录后，找到右上角的<code>my
account</code>，进入可以设置自己所在的组，这里需要设置到您所在的组，这样发送到这个组的review请求才会发送通知邮件</p>

<p><img class="center" src="http://arch.corp.anjuke.com/images/2012/10/reviewboard002.jpg"></p>

<h2>How could I get access ?</h2>

<h3>安装环境与配置</h3>

<p><strong>以下这些操作都是针对unix-like的操作系统，如果您使用的是windows，可以在本地生成diff文件后上传</strong></p>

<p>RBTools包含了一些reviewboard的自动脚本，由<a href="http://python.org">Python</a>写成，可以通过Python的包管理器<code>setuptools</code>或者<code>pip</code>安装</p>

<p>安装RBTools可以利用以下命令 (必要的时候需要sudo)</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>easy_install RBTools</span></code></pre></td></tr></table></div></figure>


<p>或者</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>pip install RBTools</span></code></pre></td></tr></table></div></figure>


<p>如果没有安装easy_install或者pip，以ubuntu为例，可以使用以下命令安装</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>aptitude install python-setuptools
</span><span class='line'>aptitude install python-pip</span></code></pre></td></tr></table></div></figure>


<p>关于Python环境，推荐使用<a href="https://github.com/utahta/pythonbrew">pythonbrew</a>，以和系统自带的Python环境隔离</p>

<p>安装完成后，会发现有多出一个<code>post-review</code>的命令，它就是我们用来和reviewboard发送review请求的工具</p>

<h2>Looks great, let&#8217;s get this thing done</h2>

<h3>设置代码仓库</h3>

<p>如果您的仓库还未在reviewboard里可用，那么需要先设置一下</p>

<p>如果您不知道怎么做，可以联系管理员(例如尔宁、Sysdev)</p>

<p>使用admin帐号登录，并选择到后台</p>

<p><img class="center" src="http://arch.corp.anjuke.com/images/2012/10/reviewboard003.jpg"></p>

<p><img class="center" src="http://arch.corp.anjuke.com/images/2012/10/reviewboard004.jpg"></p>

<p><img class="center" src="http://arch.corp.anjuke.com/images/2012/10/reviewboard005.jpg"></p>

<p>新建仓库，设置参考下图，这里的名字将会在后面<code>.reviewboardrc</code>里使用</p>

<p><img class="center" src="http://arch.corp.anjuke.com/images/2012/10/reviewboard006.jpg"></p>

<h3>配置.reviewboardrc</h3>

<p><code>post-review</code>命令会检查当前目录和用户的<code>$HOME</code>下是否有这个配置，里面可以写一些有用的信息来帮助<code>post-review</code>认识仓库和请求的一些信息</p>

<p><em><strong>推荐: 在每个代码版本仓库下都保留一个通用的<code>.reviewboardrc</code>配置</strong></em></p>

<p>一个典型的<code>.reviewboardrc</code>配置内容如下</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>REVIEWBOARD_URL="http://git.corp.anjuke.com/reviews"
</span><span class='line'>REPOSITORY='image-service'</span></code></pre></td></tr></table></div></figure>


<p><code>REVIEWBOARD_URL</code>设置了reviewboard的地址，<code>REPOSITORY</code>则是之前设置好的仓库名字</p>

<h3>发起review请求</h3>

<p>对于发送review请求，主要可以有2种形式:</p>

<ol>
<li>先发送请求，再手动上传diff文件</li>
<li>发送请求的同时上传diff文件</li>
</ol>


<h4>创建请求并手工上传diff</h4>

<p>简单的在有.reviewboardrc配置的目录下执行命令，或者在界面上新建，效果是一样的:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>post-review</span></code></pre></td></tr></table></div></figure>


<p><em><strong>注意: 第一次执行会要求输入用户名和密码；当然也可以在<code>.reviewboardrc</code>里设置</strong></em></p>

<p><img src="http://arch.corp.anjuke.com/medias/20121009/rb.howto.001.png" alt="" /></p>

<p>如上图会得到一个review请求的地址，访问它，再手工上传diff文件</p>

<p><img class="center" src="http://arch.corp.anjuke.com/images/2012/10/reviewboard007.jpg"></p>

<p>此外，还需要填入一些必要的信息</p>

<ul>
<li>summary</li>
<li>branch</li>
<li>选择group或者people</li>
<li>group的话，就和第一步准备里加入的组有关系了，组里人的才能收到邮件</li>
</ul>


<p>在这个阶段和发布后，自己以及别人就可以通过这个URL来看到您的review请求，点右上角的<code>view diff</code>可以看到具体的代码diff情况，并作出评价等</p>

<h4>生成diff的方法</h4>

<p>参考如下</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>git diff $sha1sum_1 $sha1sum_2 &gt; foobar.diff # 注意是从1到2的变化，所以别搞反了</span></code></pre></td></tr></table></div></figure>


<p>完成后点击publish即可发布这个review请求。</p>

<h3>发起请求的同时上传diff文件</h3>

<p>在git仓库里执行<code>post-review</code>时，可以通过参数直接生成diff文件并上传，命令实例:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>post-review --revision-range=970521dfb4c42f89b1a4749cec77cd919c184632:401097033760d6b188039b11aab0c853b68541b4</span></code></pre></td></tr></table></div></figure>


<p><em><strong>注意: 这里第一个版本必须在远程的代码仓库里存在，否则将无法发布review请求</strong></em></p>

<p>到生成的URL里发现已经有diff文件上传好了，只需要填入必要的字段就可以发布了。</p>

<h3>便捷的方法</h3>

<p>这里给出一个较为便捷的发起请求的方法，参数会告诉post-review来根据git的提交信息自动获取描述信息</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>post-review \
</span><span class='line'>--guess-summary \ # 自动根据git commit comment提取
</span><span class='line'>--guess-description \ # 自动根据git commit comment提取
</span><span class='line'>--revision-range=":" \ # 这里填上需要的版本号
</span><span class='line'>--branch="master" # 替换成需要的分支</span></code></pre></td></tr></table></div></figure>


<h2>Oops</h2>

<p>如果出现错误，这种时候请检查一下<code>.reviewboardrc</code>配置是否有问题；还可以通过<code>--debug</code>参数查看详细信息。</p>

<p>如果还是搞不定的话，果断求助吧!</p>

<h2>More?</h2>

<p>抱歉，目前我也只能写这么多，更多关于<code>reviewboard</code>和<code>RBTools</code>的应用请翻阅其他资料网站或者其他手册。</p>

<hr />

<p>Last updated 2012/11/27 By Lei Chen <a href="mailto:chenlei@anjuke.com">chenlei@anjuke.com</a> from
sysdev team</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[zc.buildout Workflow]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/10/02/zc-dot-buildout-workflow/"/>
    <updated>2012-10-02T14:44:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/10/02/zc-dot-buildout-workflow</id>
    <content type="html"><![CDATA[<p><img class="right" src="http://www.gravatar.com/avatar/b881be65f3c4f45dd68bcf0fbe6ba82b.png"></p>

<p>This article aims to provide a concrete workflow of using zc.buildout to deploy development and production environment.</p>

<h2>Project Structure (BuildOut SKELeton)</h2>

<p>Here is a minimum application structure with zc.buildout.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>git clone git@github.com:jizhang/boskel.git
</span><span class='line'><span class="nv">$ </span>find boskel
</span><span class='line'>boskel
</span><span class='line'>boskel/.gitignore
</span><span class='line'>boskel/bootstrap.py
</span><span class='line'>boskel/buildout.cfg
</span><span class='line'>boskel/offline.cfg
</span><span class='line'>boskel/setup.py
</span><span class='line'>boskel/src
</span><span class='line'>boskel/src/boskel
</span><span class='line'>boskel/src/boskel/__init__.py
</span></code></pre></td></tr></table></div></figure>


<h2>Development Environment</h2>

<p>To initiate the application:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span><span class="nb">cd </span>boskel
</span><span class='line'><span class="nv">$ </span>python bootstrap.py --distribute
</span><span class='line'><span class="nv">$ </span>bin/buildout
</span></code></pre></td></tr></table></div></figure>


<p>Now, a fully functional and isolated python environment has been built. Try the following commands:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>bin/python
</span><span class='line'>&gt;&gt;&gt; import boskel
</span><span class='line'>&gt;&gt;&gt;
</span></code></pre></td></tr></table></div></figure>


<p>It shows the bin/python interpreter is able to find the newly created boskel package.</p>

<p>Here are something you may want to catch up:</p>

<ul>
<li><p><strong>bootstrap.py</strong> Since not everyone has installed zc.buildout locally (by pip install zc.buildout), an initiation script is included. It will download the latest distribute package (it defaults to use setuptools, but distribute package is a replacement and better choice) and zc.buildout packages. It&#8217;ll generate bin/buildout script, which is the hero of this article.</p></li>
<li><p><strong>buildout.cfg</strong> A typical buildout configuration file. In detail, the &#8216;develop = .&#8217; tells buildout to find pakcage in the current directory, which must includes a setup.py script. And &#8216;interpreter = python&#8217; indicates that a new interpreter bin/python should be generated. This special interpreter is made capable of finding all the dependencies of this application.</p></li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>cat buildout.cfg
</span><span class='line'><span class="o">[</span>buildout<span class="o">]</span>
</span><span class='line'><span class="nv">parts</span> <span class="o">=</span> python
</span><span class='line'><span class="nv">develop</span> <span class="o">=</span> .
</span><span class='line'>
</span><span class='line'><span class="o">[</span>python<span class="o">]</span>
</span><span class='line'><span class="nv">recipe</span> <span class="o">=</span> zc.recipe.egg
</span><span class='line'><span class="nv">interpreter</span> <span class="o">=</span> python
</span><span class='line'><span class="nv">eggs</span> <span class="o">=</span>
</span><span class='line'>    boskel
</span></code></pre></td></tr></table></div></figure>


<ul>
<li><strong>setup.py</strong> A standard package setup script. &#8216;packages&#8217; and &#8216;package_dir&#8217; option should be written as is shown, since it&#8217;s a convention of buildout to put all source code in an &#8216;src&#8217; folder.</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>cat setup.py
</span><span class='line'>from setuptools import setup, find_packages
</span><span class='line'>
</span><span class='line'>setup<span class="o">(</span>
</span><span class='line'>    <span class="nv">name</span><span class="o">=</span><span class="s1">&#39;boskel&#39;</span>,
</span><span class='line'>    <span class="nv">version</span><span class="o">=</span><span class="s1">&#39;0.1.0&#39;</span>,
</span><span class='line'>    <span class="nv">packages</span><span class="o">=</span>find_packages<span class="o">(</span><span class="s1">&#39;src&#39;</span><span class="o">)</span>,
</span><span class='line'>    <span class="nv">package_dir</span><span class="o">={</span><span class="s1">&#39;&#39;</span>: <span class="s1">&#39;src&#39;</span><span class="o">}</span>,
</span><span class='line'>    <span class="nv">zip_safe</span><span class="o">=</span>False,
</span><span class='line'>    <span class="nv">install_requires</span><span class="o">=[</span>
</span><span class='line'>    <span class="o">]</span>
</span><span class='line'><span class="o">)</span>
</span></code></pre></td></tr></table></div></figure>


<ul>
<li><strong>offline.cfg</strong> This file will be used to run buildout in a remote server without internet access.</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>cat offline.cfg
</span><span class='line'><span class="o">[</span>buildout<span class="o">]</span>
</span><span class='line'><span class="nv">extends</span> <span class="o">=</span> buildout.cfg
</span><span class='line'>download-cache <span class="o">=</span> downloads
</span><span class='line'>install-from-cache <span class="o">=</span> <span class="nb">true</span>
</span></code></pre></td></tr></table></div></figure>


<h2>Production Environment</h2>

<p>When deploying to a production environment, dependency isolation is very important, unless one plans to deploy only one application per server, then he may install all the packages globally. Otherwise, an environment isolation tool should be chosen, and virtualenv is a good one.</p>

<p>A typical procedure is as below:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nb">local</span> <span class="nv">$ </span>python setup.py sdist
</span><span class='line'><span class="nb">local</span> <span class="nv">$ </span>scp xxx-0.1.tar.gz user@remote:/path/to/dist
</span><span class='line'>remote <span class="nv">$ </span><span class="nb">source </span>venv/bin/activate
</span><span class='line'>remote <span class="nv">$ </span>python setup.py install
</span></code></pre></td></tr></table></div></figure>


<p>The dependencies need to be uploaded and installed into the venv first.</p>

<p>As for buildout, we can also use the above approach, deploying with virtualenv. But what if our new distribution&#8217;s dependency tree is conflict with the previous package. Or I have to setup different virtual environments for production and staging release. So, a better solution is to use buildout for every distribution.</p>

<p>The main problem of building-out application on remote servers is dependency handling, since most of the time, there won&#8217;t be internet accessibility in those servers.</p>

<p>There&#8217;re two solutions, one is setting up a pypi repository with <a href="http://pypi.python.org/pypi/collective.eggproxy">eggproxy</a>. It will cache the eggs downloaded from internet for further use.</p>

<p>Another approach is to install from local cache, as is described below.</p>

<p>zc.buildout has an offline mode (bin/buildout -o, which is also useful when developing, since it will not try to checkout the latest packages, making the building much faster). We can trigger this behavior in configuration file, as is shown in the offline.cfg. The &#8216;download-cache&#8217; option indicates where to find the packages. Obviously there should be only one download-cache per server, and whether to use absolute path or a symbolic link to the path is your call.</p>

<p>How about the zc.buildout package itself? In development environment, we use bootstrap.py to install it, but there&#8217;s not offline-mode bootstrap.py available. So my answer to it is setting up a virtualenv which includes a zc.buildout package. To setup this venv:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span><span class="nb">cd </span>virtualenv-x.x.x
</span><span class='line'><span class="nv">$ </span>python setup.py install
</span><span class='line'><span class="nv">$ </span>virtualenv venv --distribute
</span><span class='line'><span class="nv">$ </span><span class="nb">source </span>venv/bin/activate
</span><span class='line'><span class="nv">$ </span><span class="nb">cd </span>zc.buildout-x.x.x
</span><span class='line'><span class="nv">$ </span>python setup.py install
</span></code></pre></td></tr></table></div></figure>


<p>Later, we can use the venv/bin/buildout script to run buildout (no activation needed).</p>

<p>Here&#8217;s the deploying procedure:</p>

<ol>
<li>Export the source code from git repository, and rsync it to remote server.</li>
<li>Make a symbolic link to the download cache.</li>
<li>Execute venv/bin/buildout -c offline.cfg</li>
</ol>


<p>We can use <a href="http://docs.fabfile.org/en/1.4.3/index.html">Fabric</a> to do the job. Here&#8217;s an example task:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">from</span> <span class="nn">fabric.api</span> <span class="kn">import</span> <span class="n">lcd</span><span class="p">,</span> <span class="n">local</span><span class="p">,</span> <span class="n">cd</span><span class="p">,</span> <span class="n">run</span>
</span><span class='line'><span class="kn">from</span> <span class="nn">fabric.contrib.project</span> <span class="kn">import</span> <span class="n">rsync_project</span>
</span><span class='line'>
</span><span class='line'><span class="nd">@task</span>
</span><span class='line'><span class="k">def</span> <span class="nf">deploy</span><span class="p">(</span><span class="n">rev</span><span class="o">=</span><span class="s">&#39;master&#39;</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">with</span> <span class="n">lcd</span><span class="p">(</span><span class="s">&#39;~/boskel&#39;</span><span class="p">):</span>
</span><span class='line'>        <span class="n">local</span><span class="p">(</span><span class="s">&#39;git clone --mirror git@github.com:jizhang/boskel.git&#39;</span><span class="p">)</span>
</span><span class='line'>        <span class="n">local</span><span class="p">(</span><span class="s">&#39;git remote update&#39;</span><span class="p">)</span>
</span><span class='line'>        <span class="n">rev</span> <span class="o">=</span> <span class="n">local</span><span class="p">(</span><span class="s">&#39;git rev-parse </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="n">rev</span><span class="p">)</span>
</span><span class='line'>        <span class="n">prefix</span> <span class="o">=</span> <span class="s">&#39;boskel-</span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="n">rev</span><span class="p">[:</span><span class="mi">7</span><span class="p">]</span>
</span><span class='line'>        <span class="n">local</span><span class="p">(</span><span class="s">&#39;git archive --prefix=</span><span class="si">%s</span><span class="s">/ --format=tar </span><span class="si">%s</span><span class="s"> | tar xf - -C </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span>\
</span><span class='line'>              <span class="p">(</span><span class="n">prefix</span><span class="p">,</span> <span class="n">rev</span><span class="p">,</span> <span class="s">&#39;~/boskel/dist&#39;</span><span class="p">))</span>
</span><span class='line'>
</span><span class='line'>    <span class="n">rsync_project</span><span class="p">(</span><span class="s">&#39;/path/to/boskel/dist&#39;</span><span class="p">,</span> <span class="s">&#39;~/boskel/</span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="n">prefix</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">with</span> <span class="n">cd</span><span class="p">(</span><span class="s">&#39;/path/to/boskel/dist/</span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="n">prefix</span><span class="p">):</span>
</span><span class='line'>        <span class="n">run</span><span class="p">(</span><span class="s">&#39;ln -s -n -f </span><span class="si">%s</span><span class="s"> downloads&#39;</span> <span class="o">%</span> <span class="s">&#39;/path/to/download-cache&#39;</span><span class="p">)</span>
</span><span class='line'>        <span class="n">run</span><span class="p">(</span><span class="s">&#39;/path/to/venv/bin/buildout -c offline.cfg&#39;</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<h2>Conclusion</h2>

<p>Here we combined buildout and virtualenv to setup isolated environment for every distribution. But two things need to be improved:</p>

<ol>
<li>collective.eggproxy is better than local cache.</li>
<li>It&#8217;s unnecessary to build application in every server. We should use a dedicated server to build the pakcage and rsync these files to other servers.</li>
</ol>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[A Workaround for Flask Werkzeug's Incompatibility With Buildout]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/09/08/a-workaround-for-flask-werkzeugs-incompatibility-with-buildout/"/>
    <updated>2012-09-08T21:13:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/09/08/a-workaround-for-flask-werkzeugs-incompatibility-with-buildout</id>
    <content type="html"><![CDATA[<p><img class="right" src="http://www.gravatar.com/avatar/b881be65f3c4f45dd68bcf0fbe6ba82b.png"></p>

<p>Normally, we can use the following snippet to run a flask application:</p>

<p>$ cat index.py</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
</span><span class='line'>
</span><span class='line'><span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span>
</span><span class='line'><span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">from_object</span><span class="p">(</span><span class="s">&#39;myproject.default_settings&#39;</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="nd">@app.route</span><span class="p">(</span><span class="s">&#39;/&#39;</span><span class="p">)</span>
</span><span class='line'><span class="k">def</span> <span class="nf">index</span><span class="p">():</span>
</span><span class='line'>    <span class="k">return</span> <span class="s">&#39;Hello, world!&#39;</span>
</span><span class='line'>
</span><span class='line'><span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">&#39;__main__&#39;</span><span class="p">:</span>
</span><span class='line'>    <span class="n">app</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
</span></code></pre></td></tr></table></div></figure>


<p>$ python index.py</p>

<p>But when it comes to buildout, things go wrong:</p>

<p>$ cat buildout.cfg</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="p">[</span><span class="n">buildout</span><span class="p">]</span>
</span><span class='line'><span class="n">parts</span> <span class="o">=</span> <span class="n">myproject</span>
</span><span class='line'><span class="n">develop</span> <span class="o">=</span> <span class="o">.</span>
</span><span class='line'>
</span><span class='line'><span class="p">[</span><span class="n">myproject</span><span class="p">]</span>
</span><span class='line'><span class="n">recipe</span> <span class="o">=</span> <span class="n">zc</span><span class="o">.</span><span class="n">recipe</span><span class="o">.</span><span class="n">egg</span>
</span><span class='line'><span class="n">interpreter</span> <span class="o">=</span> <span class="n">python</span>
</span><span class='line'><span class="n">eggs</span> <span class="o">=</span>
</span><span class='line'>    <span class="n">myproject</span>
</span></code></pre></td></tr></table></div></figure>


<p>$ cat src/myproject/<strong>init</strong>.py</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
</span><span class='line'>
</span><span class='line'><span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span>
</span><span class='line'><span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">from_object</span><span class="p">(</span><span class="s">&#39;myproject.default_settings&#39;</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="nd">@app.route</span><span class="p">(</span><span class="s">&#39;/&#39;</span><span class="p">)</span>
</span><span class='line'><span class="k">def</span> <span class="nf">index</span><span class="p">():</span>
</span><span class='line'>    <span class="k">return</span> <span class="s">&#39;Hello, world!&#39;</span>
</span></code></pre></td></tr></table></div></figure>


<p>$ cat index.py</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">from</span> <span class="nn">myproject</span> <span class="kn">import</span> <span class="n">app</span>
</span><span class='line'>
</span><span class='line'><span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">&#39;__main__&#39;</span><span class="p">:</span>
</span><span class='line'>    <span class="n">app</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
</span></code></pre></td></tr></table></div></figure>


<p>$ bin/python index.py</p>

<pre>
 * Running on http://127.0.0.1:5000/
 * Restarting with reloader
Traceback (most recent call last):
  File "index.py", line 1, in <module>
    from myproject import app
ImportError: No module named myproject
</pre>


<p>The problem is triggered by &#8220;app.config.from_object&#8230;&#8221;. When running this line (or a file is changed during development), Werkzeug will restart the application using the default python interpreter, not the one generated by buildout (i.e. bin/python), because bin/python removed itself from the sys.argv list while Werkzeug uses this to do reloading, which at last results in an ImportError.</p>

<p>The solution is to generate a custom script to run the server. zc.recipe.egg:scripts recipe is ready for use. We specify an &#8216;entry-points&#8217; option, like the following:
(actually scripts is the default recipe of zc.recipe.egg, so you can simply omit it)</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="p">[</span><span class="n">buildout</span><span class="p">]</span>
</span><span class='line'><span class="n">parts</span> <span class="o">=</span> <span class="n">myproject</span>
</span><span class='line'><span class="n">develop</span> <span class="o">=</span> <span class="o">.</span>
</span><span class='line'>
</span><span class='line'><span class="p">[</span><span class="n">myproject</span><span class="p">]</span>
</span><span class='line'><span class="n">recipe</span> <span class="o">=</span> <span class="n">zc</span><span class="o">.</span><span class="n">recipe</span><span class="o">.</span><span class="n">egg</span>
</span><span class='line'><span class="n">interpreter</span> <span class="o">=</span> <span class="n">python</span>
</span><span class='line'><span class="n">entry</span><span class="o">-</span><span class="n">points</span> <span class="o">=</span> <span class="n">serve</span><span class="o">=</span><span class="n">myproject</span><span class="p">:</span><span class="n">main</span>
</span><span class='line'><span class="n">eggs</span> <span class="o">=</span>
</span><span class='line'>    <span class="n">myproject</span>
</span></code></pre></td></tr></table></div></figure>


<p>And we need to add a main function to the package&#8217;s <strong>init</strong>.py:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
</span><span class='line'>    <span class="n">app</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
</span></code></pre></td></tr></table></div></figure>


<p>Execute bin/buildout again, and a script named serve will be generated in bin directory. Run it away:</p>

<p>$ bin/serve</p>

<pre>
 * Running on http://127.0.0.1:5000/
 * Restarting with reloader
</pre>


<p>It&#8217;ll work. Also the index.py is no longer necessary.</p>

<p>You can totally customize the script, like:</p>

<pre>
entry-points = script_name=project_name.module_name:function_name
</pre>


<p>For more information on zc.recipe.egg:scripts, visit:</p>

<p><a href="http://pypi.python.org/pypi/zc.recipe.egg/1.3.2#specifying-entry-points">http://pypi.python.org/pypi/zc.recipe.egg/1.3.2#specifying-entry-points</a></p>

<p>The original question can be found in the mailing list:</p>

<p><a href="http://flask.pocoo.org/mailinglist/archive/2011/7/18/flask-with-buildout/">http://flask.pocoo.org/mailinglist/archive/2011/7/18/flask-with-buildout/</a></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[深入理解daemon]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/09/02/daemon-process/"/>
    <updated>2012-09-02T13:11:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/09/02/daemon-process</id>
    <content type="html"><![CDATA[<p><img class="right" src="http://0.gravatar.com/avatar/ce1e13bbf946c92e2abf740f8909bafa"></p>

<p>用linux的各位巨巨应该都知道在系统里有种进程叫做<code>daemon</code>，一般理解为后台服务，它有一些特征，比如后台运行，不能直接在终端控制，用户退出登陆后也不会停止等等；有时候我们也想自己运行的脚本能够&#8221;后台运行&#8221;，往往使用的是<code>nohup</code>这个工具。那么daemon到底是什么呢?</p>

<p>(如果以下解释里有任何遗漏或者错误，也欢迎指出)</p>

<p>在许许多多的开源工具(例如<a href="http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/">这里</a>)里我们都能找到类似如下的代码，这2次fork被称作<code>unix magic 2 forks</code></p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="k">def</span> <span class="nf">daemonize</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
</span><span class='line'>    <span class="sd">&quot;&quot;&quot;</span>
</span><span class='line'><span class="sd">    do the UNIX double-fork magic, see Stevens&#39; &quot;Advanced</span>
</span><span class='line'><span class="sd">    Programming in the UNIX Environment&quot; for details (ISBN 0201563177)</span>
</span><span class='line'><span class="sd">    http://www.erlenstar.demon.co.uk/unix/faq_2.html#SEC16</span>
</span><span class='line'><span class="sd">    &quot;&quot;&quot;</span>
</span><span class='line'>    <span class="k">try</span><span class="p">:</span>
</span><span class='line'>          <span class="n">pid</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">fork</span><span class="p">()</span>
</span><span class='line'>        <span class="k">if</span> <span class="n">pid</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
</span><span class='line'>                <span class="c"># exit first parent</span>
</span><span class='line'>                <span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
</span><span class='line'>    <span class="k">except</span> <span class="ne">OSError</span><span class="p">,</span> <span class="n">e</span><span class="p">:</span>
</span><span class='line'>        <span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s">&quot;fork #1 failed: </span><span class="si">%d</span><span class="s"> (</span><span class="si">%s</span><span class="s">)</span><span class="se">\n</span><span class="s">&quot;</span> <span class="o">%</span> <span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">errno</span><span class="p">,</span> <span class="n">e</span><span class="o">.</span><span class="n">strerror</span><span class="p">))</span>
</span><span class='line'>        <span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>    <span class="c"># decouple from parent environment</span>
</span><span class='line'>    <span class="n">os</span><span class="o">.</span><span class="n">chdir</span><span class="p">(</span><span class="s">&quot;/&quot;</span><span class="p">)</span>
</span><span class='line'>    <span class="n">os</span><span class="o">.</span><span class="n">setsid</span><span class="p">()</span>
</span><span class='line'>    <span class="n">os</span><span class="o">.</span><span class="n">umask</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>    <span class="c"># do second fork</span>
</span><span class='line'>    <span class="k">try</span><span class="p">:</span>
</span><span class='line'>        <span class="n">pid</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">fork</span><span class="p">()</span>
</span><span class='line'>        <span class="k">if</span> <span class="n">pid</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
</span><span class='line'>            <span class="c"># exit from second parent</span>
</span><span class='line'>                <span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
</span><span class='line'>    <span class="k">except</span> <span class="ne">OSError</span><span class="p">,</span> <span class="n">e</span><span class="p">:</span>
</span><span class='line'>        <span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s">&quot;fork #2 failed: </span><span class="si">%d</span><span class="s"> (</span><span class="si">%s</span><span class="s">)</span><span class="se">\n</span><span class="s">&quot;</span> <span class="o">%</span> <span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">errno</span><span class="p">,</span> <span class="n">e</span><span class="o">.</span><span class="n">strerror</span><span class="p">))</span>
</span><span class='line'>        <span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>    <span class="c"># redirect standard file descriptors</span>
</span><span class='line'>    <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
</span><span class='line'>    <span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
</span><span class='line'>    <span class="n">si</span> <span class="o">=</span> <span class="nb">file</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">stdin</span><span class="p">,</span> <span class="s">&#39;r&#39;</span><span class="p">)</span>
</span><span class='line'>    <span class="n">so</span> <span class="o">=</span> <span class="nb">file</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">stdout</span><span class="p">,</span> <span class="s">&#39;a+&#39;</span><span class="p">)</span>
</span><span class='line'>    <span class="n">se</span> <span class="o">=</span> <span class="nb">file</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">stderr</span><span class="p">,</span> <span class="s">&#39;a+&#39;</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
</span><span class='line'>    <span class="n">os</span><span class="o">.</span><span class="n">dup2</span><span class="p">(</span><span class="n">si</span><span class="o">.</span><span class="n">fileno</span><span class="p">(),</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="o">.</span><span class="n">fileno</span><span class="p">())</span>
</span><span class='line'>    <span class="n">os</span><span class="o">.</span><span class="n">dup2</span><span class="p">(</span><span class="n">so</span><span class="o">.</span><span class="n">fileno</span><span class="p">(),</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">fileno</span><span class="p">())</span>
</span><span class='line'>    <span class="n">os</span><span class="o">.</span><span class="n">dup2</span><span class="p">(</span><span class="n">se</span><span class="o">.</span><span class="n">fileno</span><span class="p">(),</span> <span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">fileno</span><span class="p">())</span>
</span><span class='line'>
</span><span class='line'>    <span class="c"># write pidfile</span>
</span><span class='line'>    <span class="n">atexit</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">delpid</span><span class="p">)</span>
</span><span class='line'>    <span class="n">pid</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">())</span>
</span><span class='line'>    <span class="nb">file</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">pidfile</span><span class="p">,</span><span class="s">&#39;w+&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s">&quot;</span><span class="si">%s</span><span class="se">\n</span><span class="s">&quot;</span> <span class="o">%</span> <span class="n">pid</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>某始终没搞明白为什么需要fork 2次，所以带着疑问去找了找根源(网上很多解释都是一笔带过)。不过很遗憾的是文中写到的这个<a href="http://www.erlenstar.demon.co.uk/unix/faq_2.html#SEC16">来源</a>目前已经无法访问到了，幸运的是在『advanced programming in the unix enviroment』这本书里有提到，所以某去找了找(如今我已经买了一本来看XD)。</p>

<p>把关于daemon的一些特征、如何做到daemon以及稍稍探究了以下为何需要这么做，以下内容是某个人总结的。</p>

<h2>daemon 的特征和必要工作</h2>

<ul>
<li>总结引用自原书13.3 daemon process</li>
</ul>


<blockquote><p>避免不需要的交互，并且具有以下一些特征</p>

<ol>
<li>调用umask(0)，把创建文件的mask设置成0，以保证daemon本身创建的文件不会继承父进程的mask</li>
<li>调用fork，并让其父进程退出，这里有3层含义

<ul>
<li>如果父进程是一个shell命令，则退出就使得shell认为该进程已经终止</li>
<li>子进程会继承父进程的group id，从而确保自己不是group leader (后面一步setsid的先决条件) [关于这个可以参考后面一段9.5 session]</li>
<li>子进程获得一个新的进程号(PID)</li>
</ul>
</li>
<li>调用setsid，创建一个新的session，这里有3个步骤含义

<ul>
<li>子进程自己成为新session的leader进程</li>
<li>子进程自己也会成为新session里唯一一个group的leader</li>
<li>子进程没有<code>controlling terminal</code> [关于这个可以参考后面9.6 controlling terminal]</li>
</ul>
</li>
<li>更改工作路径到<code>/</code>，unix系统传统上认为一个daemon是从system boot开始一直到system halt/reboot为止一直常驻的进程，如果把工作路径设置到某个挂载上来的设备上，那么系统halt/reboot的时候设备将无法卸载(因为有进程在使用)</li>
<li>关于不需要的文件句柄(FD)，避免daemon自身打开从其父进程那里继承过来的任何FD</li>
<li>有些daemon会把FD0、FD1、FD2(分别是标准输入、标准输出和标准错误)，重定向到<code>/dev/null</code>，确保各种引用lib的流程都无效化(即没有依赖)</li>
</ol>
</blockquote>

<p>典型的来说，daemon已经不再像unix系统所阐述的那样了，一般的可以认为没有交互终端控制访问进程的后台进程都可以认为是以后宗具有daemon性质的进程，所以以上步骤中第1、4步可以选择不做或者根据需要修改。</p>

<p>我们再来看看其中一些有疑点的地方(个人来说)</p>

<h2>什么是会话(session)</h2>

<ul>
<li>总结引用自原书9.5 session</li>
</ul>


<blockquote><p>一般在进程建立新session的时候，会调用<code>setsid()</code>这个函数，它会有3个必要的步骤、特征</p>

<ol>
<li>如果调用者(以下称为caller)是一个进程组的leader，那么调用会出错，(我们可以通过fork来避免，父进程退出，子进程继续运行)

<ul>
<li>关于进程组的leader，指的就是一系列PIPE或者FORK的第一个进程，例如<code>cat foo | wc -l</code>，这里2个进程一起被称作一个group，而<code>cat</code>是这个group的leader</li>
</ul>
</li>
<li>caller本身称为一个session的leader(道理和group类似)，也是这个新session里仅有的进程

<ul>
<li>caller本身也称为一个group的leader，group id 就是caller的PID</li>
</ul>
</li>
<li>caller本身不会有<code>controlling terminal</code>，如果在<code>setsid()</code>之前就有<code>controlling terminal</code>，那么调用就会失败

<ul>
<li>关于<code>controlliing terminal</code>，可以参考下面的9.6</li>
</ul>
</li>
</ol>
</blockquote>

<h2>什么是控制终端(controlling terminal)</h2>

<ul>
<li>总结引用自原书9.6 controlling terminal</li>
</ul>


<blockquote><p>关于<code>controlling terminal</code>，有以下这些特征</p>

<ol>
<li>一个session仅有一个终端，可以是真正的终端，也可以是伪终端(例如我们常在X11里用的终端模拟器)</li>
<li>建立到<code>controlling terminal</code>的进程叫做<code>controlling process</code>，通常是<code>login shell</code>(常见的例如<code>/bin/bash --login</code>)</li>
<li>在一个session里，进程组(group)，可以分为一个前台组(foreground group)和若干个后台组(background group) (具体的可以参考unix-like系统的进程控制方面的解说)

<ul>
<li>如果有conrolling terminal的情况下，也最多只有一个前台组，其他都是后台组，如果要和controlling terminal通信，需要打开例如<code>/dev/tty</code>这样的设备</li>
</ul>
</li>
<li>键盘随时按下中断键(通常是<code>ctrl+c</code>)，会给所有前台组里的进程发送<code>SIGINT</code>的信号</li>
<li>键盘随时按下退出键(通常是<code>ctrl+bs</code>)，会给所有前台组里的进程发送<code>quit signal</code>的信号</li>
<li>如果终端接口检测到了网络断开，则会给session leader(同时也是controlling process)发送一个hang-up的信号

<ul>
<li>这也就是为什么是session leader子进程的前台组里的程序在我们断开ssh连接后会立即终止的原因</li>
</ul>
</li>
</ol>
</blockquote>

<h2>但您还未解释为什么需要fork两次?</h2>

<p>在前面解释daemon的地方，第2步fork只提到了1次，那么为什么网上那么多daemon的做法里都建议使用2次<code>fork</code>呢，原书里也做了一些解释。</p>

<ul>
<li>总结引用自原书9.6 controlling terminal以及3.3 open</li>
</ul>


<blockquote><p>一些系统在使用daemon时建议调用2次fork，情况主要是这样的</p>

<ol>
<li>基于<code>System V</code>的Unix系统会为打开一个还未分配到任何session的终端设备，把它当作<code>controlling terminal</code>来使用

<ul>
<li><code>System V</code>在调用<code>open</code>时，如果没有指定<code>0_NOCTTY</code>这个flag就会这样</li>
</ul>
</li>
<li>基于<code>BSD</code>的Unix系统会在session leader调用<code>ioctl</code>时带上<code>TIOCSCTTY</code>参数，分配一个<code>controlling terminal</code>；而如果调用时该进程恰巧已经有一个<code>controlling terminal</code>，则调用会失败，所以一般来紧跟一个<code>setsid()</code>来保证调用的正确。然后在<code>POSIX.1</code>规范中，<code>BSD</code>系统并不会使用上面提到的<code>0_NOCTTY</code>这样的参数

<ul>
<li>示例调用代码在原书19.4节</li>
</ul>
</li>
</ol>
</blockquote>

<p>这样一来就可以理解了，<u><strong>第二次调用<code>fork</code>，可以保证第二个子进程不是session leader，而之后的<code>open</code>调用，也不会分配到任何<code>controlling terminal</code>，这样就保证了daemon的必要条件。</strong></u></p>

<h2>那我们经常使用的nohup到底和daemon有什么区别?</h2>

<p>首先，请认真<code>man nohup</code> (PIA死)</p>

<p>从<a href="http://en.wikipedia.org/wiki/Nohup">维基百科的页面</a>上，我们会发现解释可以和以上这些信息结合后概括成最终下面的结论</p>

<p>可以看到在session结束后，session leader会收到hang-up的信号，而<code>nohup</code>其实时给一个进程<strong>忽略</strong> <code>SIGHUP</code>信号，<u>并非真正的daemon</u>，它仅仅提供了一种想退出登陆又不愿意中断进程的选择。</p>

<p>对于类似的功能，我个人强烈推荐tmux，以及类似功能的screen。它们甚至可以把session从controlling terminal上detach，换一个地方再attach上来，<strike><strong>是不折腾会死星人的好伙伴XDDDDDDDDD</strong></strike></p>

<h2>方便的daemon工具?</h2>

<p>而对于希望执行daemon的人来说，除了前面提到的<code>unix magic 2 forks</code>(大部分现代编程语言都提供了类似daemon这样的lib)，还可以使用debian等发行版提供的<code>/sbin/start-stop-daemon</code></p>

<p>以上</p>

<p>__END__</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Use Supervisord with Storm and ZooKeeper]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/08/27/use-supervisord-with-storm-and-zookeeper/"/>
    <updated>2012-08-27T21:25:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/08/27/use-supervisord-with-storm-and-zookeeper</id>
    <content type="html"><![CDATA[<p><img class="right" src="http://www.gravatar.com/avatar/b881be65f3c4f45dd68bcf0fbe6ba82b.png"></p>

<p>Since both &#8216;bin/storm (nimbus|supervisor|ui)&#8217; and &#8216;bin/zkServer.sh start&#8217; tend to fork a new process to start the JVM, it&#8217;s difficult to use &#8216;supervisorctl stop &#8230;&#8217; to control the processes.</p>

<p>There&#8217;re two solutions:</p>

<ol>
<li><p>Use supervisord 3.0a13 (current version is 3.0a12), which provides two options &#8216;stopasgroup, killasgroup&#8217; in [program:x] section, making it possible to stop or kill the process group. But the new release doesn&#8217;t seem to come up soon.</p></li>
<li><p>Configure the full command to supervisord. bin/storm and bin/zkServer.sh are both control scripts. The essential part is the JVM command. For Storm, when we startup one the components (nimbus, etc.), it&#8217;ll print the full command. For ZooKeeper, use &#8216;bin/zkServer.sh print-cmd&#8217;. Remember to check whether the command includes the full path to the binaries. If not, use the &#8216;directory&#8217; option in [program:x] section.</p></li>
</ol>


<p>Here is a sample configuration of supervisord.conf:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>[supervisord]
</span><span class='line'>logfile = /tmp/supervisord.log
</span><span class='line'>
</span><span class='line'>[supervisorctl]
</span><span class='line'>serverurl = unix:///tmp/supervisor.sock
</span><span class='line'>
</span><span class='line'>[unix_http_server]
</span><span class='line'>file = /tmp/supervisor.sock
</span><span class='line'>
</span><span class='line'>[rpcinterface:supervisor]
</span><span class='line'>supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
</span><span class='line'>
</span><span class='line'>[program:zookeeper]
</span><span class='line'>command = /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -Dzookeeper.log.dir="." ...
</span><span class='line'>stdout_logfile = /home/jizhang/Applications/zookeeper/zookeeper.out
</span><span class='line'>stderr_logfile = /home/jizhang/Applications/zookeeper/zookeeper.err
</span><span class='line'>autorestart = true
</span><span class='line'>
</span><span class='line'>[program:storm-nimbus]
</span><span class='line'>command = java -server -Dstorm.home=/home/jizhang/Applications/storm -Dstorm.options= -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib...
</span><span class='line'>autorestart = true
</span><span class='line'>
</span><span class='line'>[program:storm-superviosr]
</span><span class='line'>command = java -server -Dstorm.home=/home/jizhang/Applications/storm -Dstorm.options= -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib ...
</span><span class='line'>autorestart = true
</span><span class='line'>
</span><span class='line'>[program:storm-ui]
</span><span class='line'>command = java -server -Dstorm.home=/home/jizhang/Applications/storm -Dstorm.options= -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib ...
</span><span class='line'>autorestart = true</span></code></pre></td></tr></table></div></figure>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Readme驱动开发]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/07/09/readme-driven-development/"/>
    <updated>2012-07-09T14:36:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/07/09/readme-driven-development</id>
    <content type="html"><![CDATA[<p><img class="right" src="http://0.gravatar.com/avatar/ce1e13bbf946c92e2abf740f8909bafa"></p>

<p>中文翻译：<a href="https://twitter.com/AReverie">陈磊</a> AReverie<br/>
英文原文：<a href="http://tom.preston-werner.com/2010/08/23/readme-driven-development.html">Tom Preston-Werner</a></p>

<p>我最近听到很多关于测试驱动开发、行为驱动开发、极限编程和<a href="http://zh.wikipedia.org/wiki/Scrum" title="SCRUM">SCRUM</a> 、<a href="http://zh.wikipedia.org/wiki/Scrum#Scrum.E4.BC.9A.E8.AE.AE" title="站立会议">站立会议</a> 以及其他一切如何开发出更好的软件的各种方法论及技术，然而除非开发出来的软件满足使用者的需求，否则都是一纸空谈。我换种说法吧，对于错误规格的完美实现毫无价值可言。基于同样的原则，一座没有书卷而华丽修缮的图书馆也近乎无用。如果你的软件没有解决正确的问题或者根本没人知道它到底干嘛的，那可就不好玩了。</p>

<p>很好，那我们怎么解决这个问题呢？实际上比你想的要来的简单，并且其重要都足以单独列出一段。</p>

<p><strong>先写Readme</strong></p>

<p>首先，就像前面说的，在你动手写任何代码、测试、行为或者情景甚至任何东西前。别急别急，我明白我们都是程序猿，不是技术文枪手！但这就是你没想明白的地方。为一个软件写Readme着实非常有必要性。在给你的软件写点什么东西前，你根本不知道你要编写什么样的代码。在人人喊打的瀑布型设计和人见人爱的敏捷开发之间，我们似乎漏了点什么。别误会，瀑布型设计代价太大。它能把一个细致入微的庞大系统变成一个细致入微却全盘错误的系统。当初砍掉它是个正确的做法，但做法却矫枉过正。如今我们的项目里只有短小、糟糕的文档甚至完全不见踪影。你能想象有些项目完全没有Readme吗！</p>

<p>简直难以接受！在成堆的技术规格和完全没规格之间，必然会有个平衡点。那个平衡点就是朴素简明的Readme。</p>

<p>将Readme驱动开发（下文简称RDD）区别于文档驱动开发（下文简称DDD）十分重要。RDD可以理解成DDD的一个有限版本或子集。通过将你的设计文档精简成一个旨在介绍软件的单个文件，RDD能帮助你免于重蹈DDD而出现的瀑布型开发过程中出现冗长而过分细致的规格。同时，它能帮助你保持库的精简和模块化。这样一个简单的方法能帮助您免去繁琐的流程保证项目朝着正确的方向前进。</p>

<p>通过写第一个Readme，您能受益于以下这些显著的优势:</p>

<ul>
<li><p>最为重要的一点是，您给于了自己一次重新审视项目，又无需每次为了改变主意而去修改代码而花去开销的机会，例如整体应该如何组织、公用API里到底应该包含哪些东西。还记得您第一次写自动化测试代码，意识到揪出全部的错误从而防止那些问题悄悄跑进代码里去吗？在项目真正编码前写一个Reame的感觉是如出一辙的。</p></li>
<li><p>为了明确应该实现什么而去写Readme会产生一些副产品，您会得到一份写的非常棒的文档。当您在项目伊始，也就是兴奋度和动机处于最高的阶段，写一个文档会十分容易。事后再补Readme将是一种绝对的障碍，并且您会发现遗漏了许多重要的细节。</p></li>
<li><p>如果您与一群开发者一起协同工作，将会发现会从Readme收益颇多。如果其他成员在您尚未完成项目前便可以读到这份Readme，他们将大可放心的在将与您项目有交互的代码上开始工作。如果没有预先定义好的接口，团队只能顺序地编码或者面临重新实现大量代码的窘境。</p></li>
<li><p>基于白纸黑字的讨论将会容易的多，而在一个没有任何成文内容的讨论话题上，事情往往会演变成争论不休。把建议的解决方案写下来这样一个简单的行动，代表了一个具体的想法，从而能被讨论，迭代发展下去。</p></li>
</ul>


<p>把给您项目写Readme的过程当作一种真正的创造活动吧，这是您充满闪光的想法应该被表达的地方，这份文件的本意应当是阐述您的创造力和表现力。Readme应当是您代码中最为重要的一个文档，适当的做法就是首先写Readme。</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[MogileFS源码学习分享]]></title>
    <link href="http://arch.corp.anjuke.com/blog/2012/05/16/mogilefs-code-reading/"/>
    <updated>2012-05-16T10:20:00+08:00</updated>
    <id>http://arch.corp.anjuke.com/blog/2012/05/16/mogilefs-code-reading</id>
    <content type="html"><![CDATA[<p><img class="right" src="http://0.gravatar.com/avatar/78811efe9c142e0898c049686e25594d"></p>

<ul>
<li><a href="http://arch.corp.anjuke.com/blog/2012/05/16/mogilefs-code-reading-1">MogileFS启动流程</a></li>
<li><a href="http://arch.corp.anjuke.com/blog/2012/05/16/mogilefs-code-reading-2">MogileFS读取文件的过程</a></li>
<li><a href="http://arch.corp.anjuke.com/blog/2012/05/16/mogilefs-code-reading-3">Perl使用Poll多路复用</a></li>
<li><a href="http://arch.corp.anjuke.com/blog/2012/05/16/mogilefs-code-reading-4">Perl使用Epoll优化</a></li>
<li><a href="http://arch.corp.anjuke.com/blog/2012/05/16/mogilefs-code-reading-5">进程间通信【IPC】</a></li>
</ul>

]]></content>
  </entry>
  
</feed>