4 votes

S3put échoue avec "Réinitialisation de la connexion par les pairs" pour les fichiers volumineux

Je peux télécharger des petits fichiers assez bien en utilisant les mêmes commandes, cependant chaque fois que j'essaie de télécharger un fichier de 5 Go sur S3 alors le script s3 échoue :

[backup]$ s3put --bucket john-murdoch-bucket --reduced --debug 2 --callback 100 data.tgz
/usr/bin/s3put:45: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  err.message[len('No module named '):] + \
send: 'GET /?&max-keys=0 HTTP/1.1\r\nHost: john-murdoch-bucket.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nDate: Tue, 11 Jun 2013 07:00:45 GMT\r\nContent-Length: 0\r\nAuthorization: AWS =\r\nUser-Agent: Boto/2.9.4 (linux2)\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: x-amz-id-2: 
header: x-amz-request-id: 1E11AD8B3C93E6B5
header: Date: Tue, 11 Jun 2013 07:00:46 GMT
header: Content-Type: application/xml
header: Transfer-Encoding: chunked
header: Server: AmazonS3
Copying /backup/data.tgz to john-murdoch-bucket/backup/data.tgz
send: 'PUT /backup/data.tgz HTTP/1.1\r\nHost: john-murdoch-bucket.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nContent-Length: 5522978068\r\nExpect: 100-Continue\r\nx-amz-storage-class: REDUCED_REDUNDANCY\r\nDate: Tue, 11 Jun 2013 07:02:15 GMT\r\nContent-MD5: idp0nQ+9y5kshw0AjL9zMw==\r\nContent-Type: application/x-tar\r\nAuthorization: AWS =\r\nUser-Agent: Boto/2.9.4 (linux2)\r\n    \r\n'
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
Traceback (most recent call last):
  File "/usr/bin/s3put", line 374, in 
    main()
  File "/usr/bin/s3put", line 371, in main
    headers=headers)
  File "/usr/bin/s3put", line 216, in singlepart_upload
    k.set_contents_from_filename(fullpath, *kargs, **kwargs)
  File "/usr/lib/python2.6/site-packages/boto/s3/key.py", line 1241, in set_contents_from_filename
    encrypt_key=encrypt_key)
  File "/usr/lib/python2.6/site-packages/boto/s3/key.py", line 1172, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/lib/python2.6/site-packages/boto/s3/key.py", line 710, in send_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/lib/python2.6/site-packages/boto/s3/key.py", line 882, in _send_file_internal
    query_args=query_args)
  File "/usr/lib/python2.6/site-packages/boto/s3/connection.py", line 544, in make_request
    override_num_retries=override_num_retries)
  File "/usr/lib/python2.6/site-packages/boto/connection.py", line 939, in make_request
    return self._mexe(http_request, sender, override_num_retries)
  File "/usr/lib/python2.6/site-packages/boto/connection.py", line 901, in _mexe
    raise e
socket.error: [Errno 104] Connection reset by peer

Est-ce que je manque d'autres outils ou options en ligne de commande pour s3put qui me permettraient de faire plusieurs tentatives, ou de télécharger en plusieurs morceaux, ou ainsi de suite?

P.S. Je viens de faire "sudo yum upgrade python-boto" mais cela n'a rien changé.

Mise à jour - essayé s3cmd 1.0.0 :

[ec2-user@ip-10-212-234-60 backup]$ s3cmd put  data.tgz s3://john-murdoch-bucket/data.tgz
data.tgz -> s3://john-murdoch-bucket/data.tgz  [1 of 1]
      36864 of 5522978068     0% in    1s    20.31 kB/s  failed
WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
data.tgz -> s3://john-murdoch-bucket/data.tgz  [1 of 1]
     233472 of 5522978068     0% in    1s   144.89 kB/s  failed
WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
data.tgz -> s3://john-murdoch-bucket/data.tgz  [1 of 1]
      45056 of 5522978068     0% in    1s    27.88 kB/s  failed
WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
data.tgz -> s3://john-murdoch-bucket/data.tgz  [1 of 1]
      36864 of 5522978068     0% in    0s    45.52 kB/s  failed
WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.25)
WARNING: Waiting 12 sec...
data.tgz -> s3://john-murdoch-bucket/data.tgz  [1 of 1]
      20480 of 5522978068     0% in    1s    19.90 kB/s  failed
WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec...
data.tgz -> s3://john-murdoch-bucket/data.tgz  [1 of 1]
      12288 of 5522978068     0% in    2s     4.63 kB/s  failed
ERROR: Upload of 'data.tgz' failed too many times. Skipping that file.

3voto

Andrey Savov Points 31

Vous devez pip install filechunkio, puis utilisez l'option --multipart de s3put.

SistemesEz.com

SystemesEZ est une communauté de sysadmins où vous pouvez résoudre vos problèmes et vos doutes. Vous pouvez consulter les questions des autres sysadmins, poser vos propres questions ou résoudre celles des autres.

Powered by:

X