Big Data Migration Flowchart

 

 

 

 

 

 

 

★ gsutil

gsutil은 Command line에서 Cloud Storage에 액세스하는데 사용할 수 있는 Python Application이다. 

gsutil은 HTTPS 및 전송 계층 보안 (TLS)을 사용하여 업로드 및 다운로드를 포함한 모든 작업을 수행한다. 

The gsutil tool is the standard tool for small to medium sized transfers (less than 1 TB) over a typical enterprise-scale network from a private data center of from another cloud provider to Google Cloud.

 

● For transfers of larger datasets, there are two things to consider:

1. For multi-threaded transfers, use gsutil -m. Several files are processed in parallel, increasing your transfer speeds.
2. For a single large file, use Composite transfers. This method breaks large files into smaller chunks to increase transfer speed. Chunks are transferred and validated in parallel, sending all data to Google. Once the chunks arrive at Google, they are combined to form a single object. 

 

 

 

 

*** gsutil 보충 설명 (skip if you're busy) ***

● Syntax for accessing resources

prefix인 gs:// 을 사용해 Cloud Storage의 리소스를 나타낸다. 

gs:// BUCKET_NAME / OBJECT_NAME

● Create Bucket

gsutil mb -b on -l us-east1 gs://my-awesome-bucket/

● Upload an object into bucket

gsutil cp Desktop/kitten.png gs://my-awesome-bucket

● Download the object from your bucket

gsutil cp gs://my-awesome-bucket/kitten.png Desktop/kitten2.png

● List details for an object

gsutil ls -l gs://my-awesome-bucket/kitten.png

● Make the objects publicly accessible

gsutil iam ch allUsers:objectViewer gs://my-awesome-bucket

● To remove this access, use this command

gsutil iam ch -d allUsers:objectViewer gs://my-awesome-bucket

● Give someone access to your bucket

gsutil iam ch user:jane@gmail.com:objectCreator,objectViewer gs://my-awesome-bucket

● Clean up

gsutil rm -r gs://my-awesome-bucket