Centos7 build python3.8.5+scrapy+gerapy

发布时间:2020-08-27 15:53:21编辑:Run阅读(244)

    环境:CentOS Linux release 7.6.1810 (Core) mini

    更新Centos7 yum源

    yum -y install epel-release


    安装编译环境

    yum groupinstall -y "Development tools"


    更新bash和openssl漏洞以及基本软件

    yum install -y bash openssl* ntp vim wget telnet nscd


    更新服务器时间

    ntpdate ntp1.aliyun.com


    安装python3可能使用的依赖

    yum install -y openssl-devel bzip2-devel expat-devel gdbm-devel readline-devel sqlite-devel zlib zlib-devel libffi-devel

    yum install -y readline-devel.x86_64 (一定要安装这个包,不然升级完成后会发现删除退格键用不了)


    下载最新版本python

    cd /opt/

    wget https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tgz


    编译安装python3.8.5

    tar zxvf Python-3.8.5.tgz -C /usr/src/

    cd /usr/src/Python-3.8.5

    #注意:必须指定--with-ssl,否则使用pip3安装包时,访问ssl链接时,报TLS/SSL错误!

    ./configure --with-ssl --prefix=/usr/local/python38

    make && make install


    出现以下信息,表示成功安装

    Installing collected packages: setuptools, pip

      WARNING: The script easy_install-3.8 is installed in '/usr/local/python38/bin' which is not on PATH.

      Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

      WARNING: The scripts pip3 and pip3.8 are installed in '/usr/local/python38/bin' which is not on PATH.

      Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

    Successfully installed pip-20.1.1 setuptools-47.1.0


    将python38添加到系统环境变量中

    vim /etc/profile

    最后一行添加

    export PATH=$PATH:/usr/local/python38/bin


    加载环境变量

    source /etc/profile


    添加软链接

    ln -s /usr/local/python38/bin/python3.8 /usr/bin/python38

    ln -s /usr/local/python38/bin/pip3.8 /usr/bin/pip38


    升级pip

    /usr/local/python38/bin/python3.8 -m pip install --upgrade pip


    查看python版本

    python38 -V


    输出以下信息:

    Python 3.8.5


    安装gerapy

    pip38 install gerapy

    出现以下信息,表示安装成功

    Successfully installed Automat-20.2.0 MarkupSafe-1.1.1 PyDispatcher-2.0.5 PyHamcrest-2.0.2 Twisted-20.3.0 appdirs-1.4.4 apscheduler-3.5.1 attrs-20.1.0 beautifulsoup4-4.9.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 constantly-15.1.0 cryptography-3.1 cssselect-1.1.0 django-1.11.29 django-apscheduler-0.3.0 django-cors-headers-3.2.0 djangorestframework-3.9.2 furl-2.1.0 gerapy-0.9.6 gevent-20.6.2 greenlet-0.4.16 hyperlink-20.0.1 idna-2.10 incremental-17.5.0 jinja2-2.10.1 lxml-4.5.2 orderedmultidict-1.0.1 parsel-1.6.0 protego-0.1.16 pyOpenSSL-19.1.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pyee-7.0.2 pymongo-3.11.0 pymysql-0.10.0 pyppeteer-0.2.2 pyquery-1.4.1 python-scrapyd-api-2.1.2 pytz-2020.1 queuelib-1.5.0 redis-3.5.3 requests-2.24.0 scrapy-1.8.0 scrapy-redis-0.6.8 scrapy-splash-0.7.2 service-identity-18.1.0 six-1.15.0 soupsieve-2.0.1 tqdm-4.48.2 tzlocal-2.1 urllib3-1.25.10 w3lib-1.22.0 websocket-0.2.1 websockets-8.1 zope.event-4.4 zope.interface-5.1.0


    创建工作目录

    mkdir /data/gerapy_scrapy


    gerapy初始化目录

    [root@localhost data]# gerapy init /data/gerapy_scrapy/

    Initialized workspace /data/gerapy_scrapy/


    gerapy初始化数据库

    cd /data/gerapy_scrapy

    [root@localhost gerapy_scrapy]# gerapy migrate

    Operations to perform:

      Apply all migrations: admin, auth, authtoken, contenttypes, core, django_apscheduler, sessions

    Running migrations:

      Applying contenttypes.0001_initial... OK

      Applying auth.0001_initial... OK

      Applying admin.0001_initial... OK

      Applying admin.0002_logentry_remove_auto_add... OK

      Applying contenttypes.0002_remove_content_type_name... OK

      Applying auth.0002_alter_permission_name_max_length... OK

      Applying auth.0003_alter_user_email_max_length... OK

      Applying auth.0004_alter_user_username_opts... OK

      Applying auth.0005_alter_user_last_login_null... OK

      Applying auth.0006_require_contenttypes_0002... OK

      Applying auth.0007_alter_validators_add_error_messages... OK

      Applying auth.0008_alter_user_username_max_length... OK

      Applying authtoken.0001_initial... OK

      Applying authtoken.0002_auto_20160226_1747... OK

      Applying core.0001_initial... OK

      Applying core.0002_auto_20180119_1210... OK

      Applying core.0003_auto_20180123_2304... OK

      Applying core.0004_auto_20180124_0032... OK

      Applying core.0005_auto_20180131_1210... OK

      Applying core.0006_auto_20180131_1235... OK

      Applying core.0007_task_trigger... OK

      Applying core.0008_auto_20180703_2305... OK

      Applying core.0009_auto_20180711_2332... OK

      Applying core.0010_auto_20191027_2040... OK

      Applying django_apscheduler.0001_initial... OK

      Applying django_apscheduler.0002_auto_20180412_0758... OK

      Applying sessions.0001_initial... OK


    创建管理员用户,密码需要符合复杂性

    [root@localhost gerapy_scrapy]# gerapy createsuperuser

    Username (leave blank to use 'root'): admin

    Email address: admin@qq.com

    Password: 

    Password (again): 

    Superuser created successfully.


    查看selinux状态

    [root@localhost gerapy_scrapy]# sestatus

    SELinux status:                 enabled

    SELinuxfs mount:                /sys/fs/selinux

    SELinux root directory:         /etc/selinux

    Loaded policy name:             targeted

    Current mode:                   enforcing

    Mode from config file:          enforcing

    Policy MLS status:              enabled

    Policy deny_unknown status:     allowed

    Max kernel policy version:      31


    关闭selinux  重启生效

    vim /etc/selinux/config

    SELINUX=disabled


    临时生效

    setenforce 0


    为了方便直接清空iptables策略

    iptables -F


    运行gerapy

    gerapy runserver 0.0.0.0:8000


    访问gerapy:ip地址+端口

    http://192.168.18.131:8000/#/login

    image.png


    image.png


    django后台登录url(可以更改账号密码)

    http://192.168.18.131:8000/admin/login/


    image.png

关键字

上一篇: scrapy选择器css

下一篇: 没有了