linux 服务器安装s3cmd

发布时间：2019-09-13 09:28:10编辑：auto阅读（2399）

yum install python-devel python-setuptools -y
easy_install pip

报错：
[root@localhost ~]# easy_install pip
Searching for pip
Reading http://pypi.python.org/simple/pip/
Couldn't find index page for 'pip' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading http://pypi.python.org/simple/
No local packages or download links found for pip
error: Could not find suitable distribution for Requirement.parse('pip')

原因是服务器没有安装pip命令
安装pip命令解决此报错（参考博主Linux下安装pip命令）

提示：
一开始cd /usr/lib/python2.6/site-packages/
[root@localhost site-packages]# cd S3
-bash: cd: S3: 没有那个文件或目录
tar xf s3cmd.tar.gz
cd /root/s3cmd-1.6.0
pip install requests s3cmd==1.6.0

[root@localhost s3cmd-1.6.0]# pip install requests s3cmd==1.6.0

Downloading/unpacking requests
  Downloading requests-2.18.4-py2.py3-none-any.whl (88kB): 88kB downloaded
Downloading/unpacking s3cmd==1.6.0
  Downloading s3cmd-1.6.0.tar.gz (100kB): 100kB downloaded
  Running setup.py (path:/tmp/pip_build_root/s3cmd/setup.py) egg_info for package s3cmd
    Using xml.etree.ElementTree for XML processing
Downloading/unpacking certifi>=2017.4.17 (from requests)
  Downloading certifi-2017.11.5-py2.py3-none-any.whl (330kB): 330kB downloaded
Downloading/unpacking idna>=2.5,<2.7 (from requests)
  Downloading idna-2.6-py2.py3-none-any.whl (56kB): 56kB downloaded
Downloading/unpacking chardet>=3.0.2,<3.1.0 (from requests)
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB): 133kB downloaded
Downloading/unpacking urllib3>=1.21.1,<1.23 (from requests)
  Downloading urllib3-1.22-py2.py3-none-any.whl (132kB): 132kB downloaded
Downloading/unpacking python-dateutil (from s3cmd==1.6.0)
  Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194kB): 194kB downloaded
Downloading/unpacking python-magic (from s3cmd==1.6.0)
  Downloading python_magic-0.4.15-py2.py3-none-any.whl
Downloading/unpacking six>=1.5 (from python-dateutil->s3cmd==1.6.0)
  Downloading six-1.11.0-py2.py3-none-any.whl
Installing collected packages: requests, s3cmd, certifi, idna, chardet, urllib3, python-dateutil, python-magic, six
  Running setup.py install for s3cmd
    Using xml.etree.ElementTree for XML processing
    changing mode of build/scripts-2.6/s3cmd from 644 to 755
    changing mode of /usr/bin/s3cmd to 755
Successfully installed requests s3cmd certifi idna chardet urllib3 python-dateutil python-magic six
Cleaning up...

cd /usr/lib/python2.6/site-packages/S3

[root@localhost site-packages]# cd S3/

[root@localhost S3]# ls
AccessLog.py   BidirMap.py     Config.py    Crypto.py       ExitCodes.py   FileLists.py   __init__.py    PkgInfo.py    S3.py      SortedDict.py
AccessLog.pyc  BidirMap.pyc    Config.pyc   Crypto.pyc      ExitCodes.pyc  FileLists.pyc  __init__.pyc   PkgInfo.pyc   S3.pyc     SortedDict.pyc
ACL.py         CloudFront.py   ConnMan.py   Exceptions.py   FileDict.py    HashCache.py   MultiPart.py   Progress.py   S3Uri.py   Utils.py
ACL.pyc        CloudFront.pyc  ConnMan.pyc  Exceptions.pyc  FileDict.pyc   HashCache.pyc  MultiPart.pyc  Progress.pyc  S3Uri.pyc  Utils.pyc

rz -y .s3cfg 上传密钥文件到服务器root目录下。

[root@localhost ~]# ls /root/.s3cfg
/root/.s3cfg

ll /usr/lib/python2.6/site-packages/S3/Config.py
cd /usr/lib/python2.6/site-packages/S3/
cp Config.py Config.py.bak

rz -y Config.py 上传Config.py 文件到服务器/usr/lib/python2.6/site-packages/S3/ 下
cp /root/s3cmd-1.6.0/s3cmd_mult /usr/local/sbin/
cd /usr/local/sbin/
chmod +x s3cmd_mult

同步到云存储的定时脚本
[root@localhost iplog]# cat /data/web/sync.sh

#!/bin/bash
newdate=`date '+%Y-%m-%d'`
IP=`/sbin/ifconfig |grep 'inet addr' |awk -F":" '{print $2}' |awk -F'  ' '{print $1}' |awk 'NR==1{print $1}' |awk -F'.' '{a=$3"."$4;print 'a'}'`
for i in `ls /data/log`
 do 
  if [ "$i" != "$newdate" ];then 

  /usr/local/sbin/s3cmd_mult sync /data/log/"$i"/ s3://gaode/"$i"/cpc/"$IP"/ --no-check-md5 -P --parallel  && rm -rf /data/log/$i    >/dev/null 2>&1 
  fi
 done

把云端存储的日志文件目录以及目录下的文件拉取到服务器本地：

[root@localhost test001]# s3cmd sync s3://yuncongchu/2018-02-07/cpv/1.248/pv /opt/test001/ -v --no-check-md5
INFO: Retrieving list of remote files for s3://yuncongchu/2018-02-07/cpv/1.248/pv ...
INFO: Compiling list of local files...
INFO: Running stat() and reading/calculating MD5 values on 0 files, this may take some time...
INFO: Found 24 remote files, 0 local files
INFO: Verifying attributes...
INFO: Summary: 24 remote files to download, 0 local files to delete, 0 local files to hardlink
's3://yuncongchu/2018-02-07/cpv/1.248/pv/0.txt' -> '/opt/test001/pv/0.txt'  [1 of 24]
 76405655 of 76405655   100% in   43s  1713.03 kB/s  done
's3://yuncongchu/2018-02-07/cpv/1.248/pv/1.txt' -> '/opt/test001/pv/1.txt'  [2 of 24]
 56033850 of 56033850   100% in   31s  1713.76 kB/s  done
's3://yuncongchu/2018-02-07/cpv/1.248/pv/10.txt' -> '/opt/test001/pv/10.txt'  [3 of 24]
 19660800 of 80397458    24% in   11s  1703.68 kB/s^CSee ya!
[root@localhost test001]# ll /opt/test001/pv/*
-rw-r--r-- 1 apache apache 76405655 2月   7 23:59 /opt/test001/pv/0.txt
-rw-r--r-- 1 apache apache 56033850 2月   7 01:59 /opt/test001/pv/1.txt

拉取云存储的上的单个日志文件到服务器本地：

[root@localhost test001]# s3cmd get  s3://yuncongchu/2018-02-07/cpv/1.248/pv/9.txt ./9.txt
's3://yuncongchu/2018-02-07/cpv/1.248/pv/9.txt' -> './9.txt'  [1 of 1]
 75481869 of 75481869   100% in   43s  1707.12 kB/s  done
[root@localhost test001]# s3cmd get  s3://yuncongchu/2018-02-07/cpv/1.248/pv/8.txt  ./
's3://yuncongchu/2018-02-07/cpv/1.248/pv/8.txt' -> './8.txt'  [1 of 1]
 76622153 of 76622153   100% in   42s  1779.85 kB/s  done
[root@localhost test001]# ls
8.txt  9.txt  pv

查看云存储的上日志文件：

[root@localhost test001]# s3cmd ls s3://yuncongchu/2018-02-07/cpv/1.248/pv/*
2018-02-07 20:19  76405655   s3://yuncongchu/2018-02-07/cpv/1.248/pv/0.txt
2018-02-07 20:18  56033850   s3://yuncongchu/2018-02-07/cpv/1.248/pv/1.txt

[root@localhost test001]# s3cmd ls s3://yuncongchu/2018-02-07/cpv/1.248/*
                       DIR   s3://yuncongchu/2018-02-07/cpv/1.248/count/
                       DIR   s3://yuncongchu/2018-02-07/cpv/1.248/pv/

关键字：