如何在Python中查找S3分段上傳失敗的位置？

我正在實施一項 cron 作業，該作業會將大型每日備份檔案上傳到 S3 存盤桶。它大部分時間都在作業，但每隔一段時間，我會檢查一下存盤桶，檔案大小明顯小于實際大小。

它應該是大約 50GB，但上次發生時，它顯示為 34GB。我的主要問題是我不確定要嘗試/捕獲什么錯誤。

我仍然在學習 Python，所以對我來說太赤裸裸了。

from progress import ProgressPercentage  # class file progress.py
from slack import *  # function file for Slack notifications
import random
import glob
import os
import boto3
import botocore
from boto3.s3.transfer import TransferConfig

bucket = "my-s3-backup"
s3 = boto3.resource('s3')

# Grabbing the last file, and removing the full path from the string
pattern = "/path/to/backup/file/xb_*"
files = list(filter(os.path.isfile, glob.glob(pattern)))
files.sort(key=lambda x: os.path.getmtime(x))
file_to_upload = files[-1]
file_name = file_to_upload.replace('/path/to/backup/file/', '')
key_path = 'physical_db_backups/'   file_name

# Multipart upload function
def multi_part_upload():
    config = TransferConfig(multipart_threshold=1024 * 25,
                            max_concurrency=10,
                            multipart_chunksize=1024 * 25,
                            use_threads=True)

    try:
        s3.meta.client.upload_file(file_to_upload, bucket, key_path, Config=config,
                                   Callback=ProgressPercentage(file_to_upload))

        # Custom Slack notification to inform completion
        sendslacksuccess("Physical Backup to S3 Complete:\n"   file_name)
    except botocore.exceptions.ClientError as error:
        
        # Custom Slack notification to inform of failure
        sendslackerror("Physical Backup to S3 Failed:\n"   file_name   "\nError: "   error)


if __name__ == '__main__':
    multi_part_upload()

如果腳本沒有“失敗”，但它沒有上傳完整的檔案大小，我想在這里捕捉什么例外？我應該在某處記錄輸出嗎？

我正在查看Botocore Exceptions檔案。我只是不確定該怎么辦try/catch。

作為參考，這里是檔案大小差異：

aws s3 ls --summarize --human-readable --recursive s3://my-s3-backup/physical_db_backups/

2022-05-07 14:31:28   50.7 GiB physical_db_backups/xb_202205070101.xb.zst
2022-05-08 12:48:07   50.8 GiB physical_db_backups/xb_202205080101.xb.zst
2022-05-09 01:30:04   34.2 GiB physical_db_backups/xb_202205090101.xb.zst <--- WRONG

uj5u.com熱心網友回復：

好吧，因為我是個白癡，沒有意識到檔案還沒有完成，所以我做了一些更改。

1. 我編輯了 cron 以便稍后開始。

2. 我創建了邏輯來確定備份腳本是否正在運行。

3. 我可能會合并額外的檢查以確保檔案存在，但現在這是一個已經過測驗的作業 POC。

    from progress import ProgressPercentage  # class file progress.py
    from slack import *  # function file for Slack notifications
    import random
    from time import sleep
    import psutil
    import glob
    import os
    import boto3
    import botocore
    from boto3.s3.transfer import TransferConfig
    import logging
   
    bucket = "fsn-s3-backup"
    s3 = boto3.resource('s3')
    pattern = "/path/to/backup/file/xb_*"
    files = list(filter(os.path.isfile, glob.glob(pattern)))
    files.sort(key=lambda x: os.path.getmtime(x))
    file_to_upload = files[-1]
    file_name = file_to_upload.replace('/path/to/backup/file/', '')
    key_path = 'physical_db_backups/'   file_name
   
    logging.basicConfig(filename='/var/log/s3-backup.log', format='%(asctime)s - %(levelname)s - %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p', filemode='a')
    logger = logging.getLogger()
    logger.setLevel(logging.INFO)
   
   
    def multi_part_upload():
        config = TransferConfig(multipart_threshold=1024 * 25,
                                max_concurrency=10,
                                multipart_chunksize=1024 * 25,
                                use_threads=True)
   
        try:
            s3.meta.client.upload_file(file_to_upload, bucket, key_path, Config=config,
                                       Callback=ProgressPercentage(file_to_upload),
                                       ExtraArgs={'ContentType': 'application/zstd'})
            logger.info("Physical Backup to S3 Complete")
            sendslacksuccess("Physical Backup to S3 Complete:\n"   file_name)
        except botocore.exceptions.ClientError as error:
            logger.error("Physical Backup to S3 Failed: "   error)
            sendslackerror("Physical Backup to S3 Failed:\n"   file_name   "\nError: "   error)
   
   
    def checkIfProcessRunning(processName):
        for proc in psutil.process_iter():
            cmdline = proc.cmdline()
            if processName in cmdline:
                return True
        return False
   
   
    if __name__ == '__main__':
        backuprunning = True
        while backuprunning:
            logger.info("Checking if backup shell script is running")
            if checkIfProcessRunning('/path/to/physical_backup.sh'):
                logger.info("Backup shell script still running. Sleeping for 60s")
                sleep(60)
            else:
                backuprunning = False
                logger.info("Beginning multipart upload")
                multi_part_upload()
   

標籤：python-3.x 亚马逊-s3 博托3 博托核

上一篇：上傳的S3檔案已損壞

下一篇：如何將所有檔案從一個檔案夾復制到同一個S3存盤桶中的另一個檔案夾