您的位置:首页 > 运维架构 > Apache

pip2pi和pypiserver及Apache在pip本地源配置中的应用实践

2015-07-28 22:19 931 查看
pip介绍:

pip是python比较流行和强大的包管理工具,pip即将替代setuptools(setuptools也可以看成easy_install的替代词,安装setuptools之后

将使用easy_install命令行来安装包),但是,pip依然是建立在setuptools基础之上的,安装pip之前必须安装setuptools。

pip有强大:

Usage:   

  pip [options]

Commands:

 install                    Install packages.

 uninstall                  Uninstall packages.

 freeze                     Output installed packages in requirements format.

 list                       List installed packages.

 show                       Show information about installed packages.

 search                     Search PyPI for packages.

 wheel                      Build wheels from your requirements.

 help                       Show help for commands.

General Options:

  -h,--help                 Show help.

 --isolated                 Run pip in an isolated mode, ignoring environment variables anduser configuration.

  -v,--verbose              Give more output. Option is additive, and can be used up to 3times.

  -V,--version              Show version and exit.

  -q,--quiet                Give less output.

  --log               Path to a verbose appending log.

  --proxy            Specify a proxy in the form [user:passwd@]proxy.server:port.

  --retries        Maximum number of retries each connection should attempt (default 5times).

  --timeout            Set the socket timeout (default 15 seconds).

  --exists-action    Defaultaction when a path already exists: (s)witch, (i)gnore, (w)ipe,(b)ackup.

  --trusted-host   Mark this host as trusted,even though it does not have valid or any HTTPS.

  --cert              Path to alternate CA bundle.

  --client-cert       Path to SSL client certificate, a single file containing theprivate key and the certificate in PEM format.

  --cache-dir          Store the cache data in.

 --no-cache-dir             Disable the cache.

  --disable-pip-version-check

                             Don't periodically check PyPI to determine whether a new version ofpip is available for download. Implied with --no-index.

pip默认的源(index-pypi=http://pypi.python.org/simple),pip install 和pipserach命令默认都会到pip.python.org上搜索下载pypi包

pypi的一个强大功能就是:把应用全部的依赖包及其版本全部写入一个requirements.txt的文件中,然后通过pipinstall -r requirements.txt

依次按照requirements.txt中的顺序安装pypi包。pip在安装过程中,默认的安装文件缓存在HOME/.cache/pip目录里面,改目录下回生产1,2,3....,a,b,c

的子目录用以存放缓存文件。因此,如果想要再次重新安装可以不必通过外网,通过如下命令:pip install pypi_nmae--src $HOME/.cache或者pip install -r requirements.txt --src$HOME/.cache

既可以通过本地cache目录进行安装。

[root@kilo http]# pwd

/root/.cache/pip/http

[root@kilo http]# ll

total 0

drwx------.  9 root root 62 Jul 27 16:22 0

drwx------.  6 root root 38 Jul 27 16:23 1

drwx------. 13 root root 94 Jul 27 16:23 2

drwx------. 11 root root 78 Jul 27 16:22 3

drwx------. 13 root root 94 Jul 27 16:22 4

drwx------. 10 root root 70 Jul 27 16:23 5

drwx------. 12 root root 86 Jul 27 16:22 6

drwx------. 11 root root 78 Jul 27 16:22 7

drwx------.  9 root root 62 Jul 27 16:22 8

drwx------. 13 root root 94 Jul 27 16:23 9

drwx------. 12 root root 86 Jul 27 16:21 a

drwx------. 13 root root 94 Jul 27 16:23 b

drwx------. 11 root root 78 Jul 27 16:23 c

drwx------. 11 root root 78 Jul 27 16:23 d

drwx------.  9 root root 62 Jul 27 16:23 e

drwx------. 11 root root 78 Jul 27 16:17 f

pip本地源的搭建:

pip本地源搭建的思想就是:先下载pypi包,然后指定pip本地源

此处介绍两种方法:

1:新建一个目录用以存放pypi包,如/root/pypi

2.将所有依赖合并到一个文件中:如 find . -name requirements.txt -exec cat {} \;>pip_requirement.all

3.开始下载pypi包到指定目录:

#!/bin/bash

PIP_REQUIRE="pip_requirement.all"

while read LINE

do

  if [[ $LINE =~ ^[a-zA-Z] ]]

  then

    echo$LINE

    pip install$LINE -d /root/pypi  #仅下载不安装

  fi

done < $PIP_REQUIRE

此步骤将会把pip_requirement.all中指定的全部依赖包下载到/root/pypi目录中。在下载过程中,默认回到官网pypi.python.org

去下载,GFW导致下载较慢,可以采用国内豆瓣源来下载,非常快相对,具体方法:

编辑 .pip/pip.conf文件,加入如下语句:

[global]

trusted-host = pypi.douban.com

index-url = http://pypi.douban.com/simple
或者采用阿里云源:

[global]

trusted-host=mirrors.aliyun.com

index-url=http://mirrors.aliyun.com/pypi/simple/

语句trusted-host=xxx是因为pip新版本为认为http未不信任主机,不让你连接。

4.更简单的批量pypi包下载方式是安装pip2pi工具:pip install pip2pi

下载单个包:pip2tgz /save/to/path pypi_nmae

批量下载:pip2tgz /save/to/path -r requirements.txt

采用此方式将会把requirements.txt里面全部pypi包下载到指定的目录(如果采用Apache部署pip源,推荐奖此路径设为Apache的默认主目录:

/var/www/html/pypi,如pip2tgz /var/www/html/pypi -r/root/openstack/keystone/requirements.txt)

5.生成pypi的index:dir2pi --normalize-package-names/var/www/html/pypi.命令将会在pypi里面生成simple文件,simple中的文件名是标准化后的

packeage名:

[root@kilo pypi]# cd /var/www/html/pypi/simple

[root@kilo simple]# ll

total 8

drwxr-xr-x. 2 root root   62 Jul28 17:53 aioeventlet

drwxr-xr-x. 2 root root   50 Jul28 17:53 alembic

drwxr-xr-x. 2 root root   57 Jul28 17:53 amqp

drwxr-xr-x. 2 root root   50 Jul28 17:53 anyjson

drwxr-xr-x. 2 root root   64 Jul28 17:53 appdirs

drwxr-xr-x. 2 root root   65 Jul28 17:53 argparse

drwxr-xr-x. 2 root root   46 Jul28 17:53 babel

drwxr-xr-x. 2 root root   60 Jul28 17:53 bandit

drwxr-xr-x. 2 root root   60 Jul28 17:53 bashate

drwxr-xr-x. 2 root root   67 Jul28 17:53 beautifulsoup4

drwxr-xr-x. 2 root root   67 Jul28 17:53 cachetools

drwxr-xr-x. 2 root root   47 Jul28 17:53 cffi

drwxr-xr-x. 2 root root   54 Jul28 17:53 contextlib2

simple目录中的每一个子目录都将生成对应的index.html文件,文件内容如:WebOb-1.4.1.tar.gz

6.配置Apache服务器。

Apache服务器主要是配置/etc/httpd/conf/httpd.conf文件,就本文情况,不要修改默认的DocumentRoot"/var/www/html"

directory字段添加 Options Indexes MultiViews FollowSymlinks

Indexes:指明在没有index.html的情况下允许列出文件和目录;

MultiViews:允许根据内容情况多种视图列出

FollowSymlinks:允许符号连接

把deny的地方改成 :Require all granted

重启Apache:systemctl restart httpd.service

查看Apache监听情况:netst
4000
at -ntl |grep 80

浏览器输入:http://host_ip/pypi/simple ,页面将以超链接的形式显示全部pypi包:





任意点击即可下载!

7.设置pip本地源。

可以使用如下形式安装包:pip install --index-url http://host_ip/pypi/simple -rrequirements.txt

不过比较麻烦,可以修改~/.pip/pip.conf来指定默认本地pip源

[global]

trusted-host = host_ip

index-url = http://host_ip/pypi/simple
如此,即可以采用本地pip源安装:pip install -r requirements.txt

将会发现速度非常快,比豆瓣源要快!此外,如果需要批量下载pypi包,如下:pip uninstall -rrequirements.txt -y

-y 参数表示无需确认即卸载。

另外一种搭建pip本地源的方式是采用pypiserver包来搭建:pip install pypiserver

pypi-server -h

pypi-server [OPTIONS] [PACKAGES_DIRECTORY...]

  start PyPI compatible package server servingpackages from

  PACKAGES_DIRECTORY. If PACKAGES_DIRECTORY is notgiven on the

  command line, it uses the default~/packages.  pypiserver scans this

  directory recursively for packages. It skipspackages and

  directories starting with a dot. Multiplepackage directories can be

  specified.

pypi-server understands the following options:

  -p, --port PORT

    listen onport PORT (default: 8080)

  -i, --interface INTERFACE

    listen oninterface INTERFACE (default: 0.0.0.0, any interface)

  -a, --authenticate (UPDATE|download|list),...

   comma-separated list of (case-insensitive) actions toauthenticate

    (requiresgiving also the -P option). For example to password-protect

    packageuploads & downloads while leaving listings public, give:

     -a update,download.

    By default,only 'update' is password-protected.

  -P, --passwords PASSWORD_FILE

    use apachehtpasswd file PASSWORD_FILE to set usernames & passwords

    used forauthentication of certain actions (see -a option).

  --disable-fallback

    disableredirect to real PyPI index for packages not found in the

    localindex

  --fallback-url FALLBACK_URL

    for packagesnot found in the local index, this URL will be used to

    redirect to(default: http://pypi.python.org/simple)
  --server METHOD

    use METHODto run the server. Valid values include paste,

    cherrypy,twisted, gunicorn, gevent, wsgiref, auto. The

    default isto use "auto" which chooses one of paste, cherrypy,

    twisted orwsgiref.

  -r, --root PACKAGES_DIRECTORY

    [deprecated]serve packages from PACKAGES_DIRECTORY

  -o, --overwrite

    allowoverwriting existing package files

  --welcome HTML_FILE

    uses theASCII contents of HTML_FILE as welcome message response.

  -v

    enableverbose logging;  repeate for moreverbosity.

  --log-file

    writelogging info into this FILE.

  --log-frmt

    the loggingformat-string.  (see `logging.LogRecord` classfrom standard python library)

    [Default:%(asctime)s|%(levelname)s|%(thread)d|%(message)s]

  --log-req-frmt FORMAT

    aformat-string selecting Http-Request properties to log; setto  '%s' to see them all.

    [Default:%(bottle.request)s]

    

  --log-res-frmt FORMAT

    aformat-string selecting Http-Response properties to log; setto  '%s' to see them all.

    [Default:%(status)s]

  --log-err-frmt FORMAT

    aformat-string selecting Http-Error properties to log; setto  '%s' to see them all.

    [Default:%(body)s: %(exception)s

%(traceback)s]

  --cache-control AGE

    Add"Cache-Control: max-age=AGE, public" header to packagedownloads.

    Pip 6+ needsthis for caching.

pypi-server -h

pypi-server --help

  show this help message

pypi-server --version

  show pypi-server's version

pypi-server -U [OPTIONS] [PACKAGES_DIRECTORY...]

  update packages in PACKAGES_DIRECTORY. Thiscommand searches

  pypi.python.org for updates and shows a pipcommand line which

  updates the package.

The following additional options can be specified with -U:

  -x

    execute thepip commands instead of only showing them

  -d DOWNLOAD_DIRECTORY

    downloadpackage updates to this directory. The default is to use

    thedirectory which contains the latest version of the package to

    beupdated.

  -u

    allowupdating to unstable version (alpha, beta, rc, dev versions)

Visit https://pypi.python.org/pypi/pypiserver for moreinformation.

pypi-server的配置相对简单,不需要再去配置apache服务。

总结起来,pip本地源的配置大概两种方式:

1.pip2pi + apache

2.pip2pi + pypiserver

第一张方式即是利用pip2pi的pip2tgz -r requirements.txt命令批量下载python包,然后利用dir2pipackges_dir 生成index,然后配置

Apache服务器,通过http访问本地repos来搭建本地源;

第二章方式在下载python包部分也是利用pip2pi命令来下载,然后将存放python包的目录作为pypiserver的包仓库,通过启动pypiserver服务来

监听端口:pypi-server -i host_ip -p port packges_dir1 packges_dir12 ...packges_dir1n &

通过netstat -ntl可以查看端口监听状态,通过lsof -i:port 可以看到监听端口的应用程序名称和进程ID

pypi-server不需要使用dir2pi生成index,只需要将需要的python包下载到本地,然后启动pypiserver监听即可!

pypi-server可以使用--fall-back 参数指定在找不到需要的包时,pip到什么地方去搜索包,如:

pypi-server -i 192.168.142.10 -p 8080 --fallback-urlhttp://192.168.142.10/pypi/simple/ pypi &

pypiserver监听192.168.142.10:8080端口,同时使用url=http://192.168.142.10/pypi/simple/作为候选index-url(此列为前面配置的

Apacheindex),如果在~/pypi目录中找不到需要的python包,则到http://192.168.142.10/pypi/simple中去找。

启动pypiserver监听之后,为了便于安装,对~.pip/pip.conf进行配置:

[global]

##############douban mirrors###############

#trusted-host=pypi.douban.com

#index-url = http://pypi.douban.com/simple
##############pip2pi&apache local repos##############

#trusted-host=192.168.142.10

#index-url = http://192.168.142.10/pypi/simple
################pypiserver local repos################

trusted-host=192.168.142.10

index-url = http://192.168.142.10:8080/simple/
完成之后,即可pip install -r requirements.txt 进行安装!

注:如果要停止pypiserver,需要lsof -i:8080 查看到进程ID ,然后kill PID!
内容来自用户分享和网络整理,不保证内容的准确性,如有侵权内容,可联系管理员处理 点击这里给我发消息
标签:  pip python pypi