bash - Uploading files to s3 using s3cmd in parallel -

bash - Uploading files to s3 using s3cmd in parallel -

i've got whole heap of files on server, , want upload these onto s3. files stored .data extension, they're bunch of jpegs,pngs,zips or pdfs.

i've written short script finds mime type , uploads them onto s3 , works it's slow. there way create below run using gnu parallel?

#!/bin/bash n in $(find -name "*.data") data=".data" extension=`file $n | cutting -d ' ' -f2 | awk '{print tolower($0)}'` mimetype=`file --mime-type $n | cutting -d ' ' -f2` fullpath=`readlink -f $n` changed="${fullpath/.data/.$extension}" filepathwithextensionchanged=${changed#*internal_data} s3upload="s3cmd set -m $mimetype --acl-public $fullpath s3://tff-xenforo-data"$filepathwithextensionchanged response=`$s3upload` echo $response done

also i'm sure code improved in general :) feedback tips appreciated.

you skilled in writing shell, , extremely close solution:

s3upload_single() { n=$1 data=".data" extension=`file $n | cutting -d ' ' -f2 | awk '{print tolower($0)}'` mimetype=`file --mime-type $n | cutting -d ' ' -f2` fullpath=`readlink -f $n` changed="${fullpath/.data/.$extension}" filepathwithextensionchanged=${changed#*internal_data} s3upload="s3cmd set -m $mimetype --acl-public $fullpath s3://tff-xenforo-data"$filepathwithextensionchanged response=`$s3upload` echo $response } export -f s3upload_single find -name "*.data" | parallel s3upload_single

bash amazon-s3 parallel-processing s3cmd gnu-parallel