gcloud beta composer environments storage data import

gcloud beta composer environments storage data import - import data from local storage or Cloud Storage into an environment

gcloud beta composer environments storage data import --source=SOURCE (--environment=ENVIRONMENT : --location=LOCATION) [--destination=DESTINATION] [GCLOUD_WIDE_FLAG …]

foo
|
+-- subdir1
|   |
|   +-- file1.txt
|   +-- file2.txt
|
+-- subdir2
|   |
|   +-- file3.txt
|   +-- file4.txt

And the environment myenv's Cloud Storage bucket has the following structure:

gs://the-bucket
|
+-- data
|   |
|   +-- foo
|   |   |
|   |   +-- subdir1
|   |   |   |
|   |   |   +-- bar.txt

The following command:

gcloud beta composer environments storage data import myenv --source=/foo

would result in the following structure in myenv's Cloud Storage bucket:

gs://the-bucket
|
+-- data
|   |
|   +-- foo
|   |   |
|   |   +-- subdir1
|   |   |   |
|   |   |   +-- bar.txt
|   |   |   +-- file1.txt
|   |   |   +-- file2.txt
|   |   |
|   |   +-- subdir2
|   |   |   |
|   |   |   +-- file3.txt
|   |   |   +-- file4.txt

If instead we had run

gcloud beta composer environments storage data import myenv --source=/foo --destination=bar

the resulting bucket structure would be the following:

gs://the-bucket
|
+-- data
|   |
|   +-- foo
|   |   |
|   |   +-- subdir1
|   |   |   |
|   |   |   +-- bar.txt
|   |
|   +-- bar
|   |   |
|   |   +-- foo
|   |   |   |
|   |   |   +-- subdir1
|   |   |   |   |
|   |   |   |   +-- file1.txt
|   |   |   |   +-- file2.txt
|   |   |   |
|   |   |   +-- subdir2
|   |   |   |   |
|   |   |   |   +-- file3.txt
|   |   |   |   +-- file4.txt

--source=SOURCE

Path to a local directory/file or Cloud Storage bucket/object to be imported into the data/ subdirectory in the environment's Cloud Storage bucket. Cloud Storage paths must begin with 'gs://'.

Environment resource - The environment into whose Cloud Storage bucket to import data.. The arguments in this group can be used to specify the attributes of this resource. (NOTE) Some attributes are not given arguments in this group but can be set in other ways.

To set the project attribute:

• provide the argument --environment on the command line with a fully specified name;
• provide the argument --project on the command line;
• set the property core/project.

This must be specified.

--environment=ENVIRONMENT

ID of the environment or fully qualified identifier for the environment.

To set the environment attribute:

• provide the argument --environment on the command line.

This flag argument must be specified if any of the other arguments in this group are specified.

--location=LOCATION

Region where Composer environment runs or in which to create the environment.

To set the location attribute:

• provide the argument --environment on the command line with a fully specified name;
• provide the argument --location on the command line;
• set the property composer/location.

--destination=DESTINATION

An optional subdirectory under the data/ directory in the environment's Cloud Storage bucket into which to import files. May contain forward slashes to delimit multiple levels of subdirectory nesting, but should not contain leading or trailing slashes. If the DESTINATION does not exist, it will be created.

Run $ gcloud help for details.