Centos7 build python3.8.5+scrapy+gerapy

发布时间：2020-08-27 15:53:21编辑：Run阅读（4143）

环境:CentOS Linux release 7.6.1810 (Core) mini


更新Centos7 yum源

yum -y install epel-release




安装编译环境

yum groupinstall -y "Development tools"




更新bash和openssl漏洞以及基本软件

yum install -y bash openssl* ntp vim wget telnet nscd




更新服务器时间

ntpdate ntp1.aliyun.com




安装python3可能使用的依赖

yum install -y openssl-devel bzip2-devel expat-devel gdbm-devel readline-devel sqlite-devel zlib zlib-devel libffi-devel

yum install -y readline-devel.x86_64 (一定要安装这个包，不然升级完成后会发现删除退格键用不了)




下载最新版本python

cd /opt/

wget https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tgz




编译安装python3.8.5

tar zxvf Python-3.8.5.tgz -C /usr/src/

cd /usr/src/Python-3.8.5

#注意：必须指定--with-ssl，否则使用pip3安装包时，访问ssl链接时，报TLS/SSL错误！

./configure --with-ssl --prefix=/usr/local/python38

make && make install




出现以下信息，表示成功安装

Installing collected packages: setuptools, pip

  WARNING: The script easy_install-3.8 is installed in '/usr/local/python38/bin' which is not on PATH.

  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

  WARNING: The scripts pip3 and pip3.8 are installed in '/usr/local/python38/bin' which is not on PATH.

  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

Successfully installed pip-20.1.1 setuptools-47.1.0




将python38添加到系统环境变量中

vim /etc/profile

最后一行添加

export PATH=$PATH:/usr/local/python38/bin




加载环境变量

source /etc/profile




添加软链接

ln -s /usr/local/python38/bin/python3.8 /usr/bin/python38

ln -s /usr/local/python38/bin/pip3.8 /usr/bin/pip38




升级pip

/usr/local/python38/bin/python3.8 -m pip install --upgrade pip




查看python版本

python38 -V




输出以下信息：

Python 3.8.5




安装gerapy

pip38 install gerapy

出现以下信息,表示安装成功

Successfully installed Automat-20.2.0 MarkupSafe-1.1.1 PyDispatcher-2.0.5 PyHamcrest-2.0.2 Twisted-20.3.0 appdirs-1.4.4 apscheduler-3.5.1 attrs-20.1.0 beautifulsoup4-4.9.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 constantly-15.1.0 cryptography-3.1 cssselect-1.1.0 django-1.11.29 django-apscheduler-0.3.0 django-cors-headers-3.2.0 djangorestframework-3.9.2 furl-2.1.0 gerapy-0.9.6 gevent-20.6.2 greenlet-0.4.16 hyperlink-20.0.1 idna-2.10 incremental-17.5.0 jinja2-2.10.1 lxml-4.5.2 orderedmultidict-1.0.1 parsel-1.6.0 protego-0.1.16 pyOpenSSL-19.1.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pyee-7.0.2 pymongo-3.11.0 pymysql-0.10.0 pyppeteer-0.2.2 pyquery-1.4.1 python-scrapyd-api-2.1.2 pytz-2020.1 queuelib-1.5.0 redis-3.5.3 requests-2.24.0 scrapy-1.8.0 scrapy-redis-0.6.8 scrapy-splash-0.7.2 service-identity-18.1.0 six-1.15.0 soupsieve-2.0.1 tqdm-4.48.2 tzlocal-2.1 urllib3-1.25.10 w3lib-1.22.0 websocket-0.2.1 websockets-8.1 zope.event-4.4 zope.interface-5.1.0




创建工作目录

mkdir /data/gerapy_scrapy




gerapy初始化目录

[root@localhost data]# gerapy init /data/gerapy_scrapy/

Initialized workspace /data/gerapy_scrapy/




gerapy初始化数据库

cd /data/gerapy_scrapy

[root@localhost gerapy_scrapy]# gerapy migrate

Operations to perform:

  Apply all migrations: admin, auth, authtoken, contenttypes, core, django_apscheduler, sessions

Running migrations:

  Applying contenttypes.0001_initial... OK

  Applying auth.0001_initial... OK

  Applying admin.0001_initial... OK

  Applying admin.0002_logentry_remove_auto_add... OK

  Applying contenttypes.0002_remove_content_type_name... OK

  Applying auth.0002_alter_permission_name_max_length... OK

  Applying auth.0003_alter_user_email_max_length... OK

  Applying auth.0004_alter_user_username_opts... OK

  Applying auth.0005_alter_user_last_login_null... OK

  Applying auth.0006_require_contenttypes_0002... OK

  Applying auth.0007_alter_validators_add_error_messages... OK

  Applying auth.0008_alter_user_username_max_length... OK

  Applying authtoken.0001_initial... OK

  Applying authtoken.0002_auto_20160226_1747... OK

  Applying core.0001_initial... OK

  Applying core.0002_auto_20180119_1210... OK

  Applying core.0003_auto_20180123_2304... OK

  Applying core.0004_auto_20180124_0032... OK

  Applying core.0005_auto_20180131_1210... OK

  Applying core.0006_auto_20180131_1235... OK

  Applying core.0007_task_trigger... OK

  Applying core.0008_auto_20180703_2305... OK

  Applying core.0009_auto_20180711_2332... OK

  Applying core.0010_auto_20191027_2040... OK

  Applying django_apscheduler.0001_initial... OK

  Applying django_apscheduler.0002_auto_20180412_0758... OK

  Applying sessions.0001_initial... OK




创建管理员用户,密码需要符合复杂性

[root@localhost gerapy_scrapy]# gerapy createsuperuser

Username (leave blank to use 'root'): admin

Email address: admin@qq.com

Password: 

Password (again): 

Superuser created successfully.




查看selinux状态

[root@localhost gerapy_scrapy]# sestatus

SELinux status:                 enabled

SELinuxfs mount:                /sys/fs/selinux

SELinux root directory:         /etc/selinux

Loaded policy name:             targeted

Current mode:                   enforcing

Mode from config file:          enforcing

Policy MLS status:              enabled

Policy deny_unknown status:     allowed

Max kernel policy version:      31




关闭selinux  重启生效

vim /etc/selinux/config

SELINUX=disabled




临时生效

setenforce 0




为了方便直接清空iptables策略

iptables -F




运行gerapy

gerapy runserver 0.0.0.0:8000




访问gerapy:ip地址+端口

http://192.168.18.131:8000/#/login











django后台登录url(可以更改账号密码)

http://192.168.18.131:8000/admin/login/

关键字：