Skip to content

Commit

Permalink
Local filesystem to Google Drive Operator - system tests migration (A…
Browse files Browse the repository at this point in the history
…IP-47) (#26797)
  • Loading branch information
bkossakowska committed Oct 10, 2022
1 parent b4cef6d commit b54a2de
Show file tree
Hide file tree
Showing 6 changed files with 43 additions and 8 deletions.
3 changes: 1 addition & 2 deletions docs/apache-airflow-providers-google/example-dags.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,12 @@
Example DAGs
============

You can learn how to use Google integrations by analyzing the source code of the example DAGs:

* `Google Ads <https://github.com/apache/airflow/tree/providers-google/8.1.0/tests/system/providers/google/ads>`__
* `Google Cloud (legacy) <https://github.com/apache/airflow/tree/providers-google/8.0.0/airflow/providers/google/cloud/example_dags>`__
* `Google Cloud <https://github.com/apache/airflow/tree/providers-google/8.0.0/tests/system/providers/google>`__
* `Google Firebase <https://github.com/apache/airflow/tree/providers-google/8.1.0/tests/system/providers/google/firebase>`__
* `Google Marketing Platform <https://github.com/apache/airflow/tree/providers-google/8.0.0/airflow/providers/google/marketing_platform/example_dags>`__
* `Google Workplace <https://github.com/apache/airflow/tree/providers-google/8.0.0/airflow/providers/google/suite/example_dags>`__ (formerly Google Suite)
* `Google Workplace <https://github.com/apache/airflow/tree/providers-google/8.3.0/tests/system/providers/google/suite>`__ (formerly Google Suite)
* `Google LevelDB <https://github.com/apache/airflow/tree/providers-google/8.0.0/tests/system/providers/google/leveldb>`__
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,11 @@ LocalFilesystemToGoogleDriveOperator
data from local filesystem to GoogleDrive.

When you use this operator, you can upload a list of files.

Below is an example of using this operator to upload data from local filesystem to Google Drive.

.. exampleinclude:: /../../tests/system/providers/google/suite/example_local_to_drive.py
:language: python
:dedent: 0
:start-after: [START howto_operator_local_to_drive_upload_single_file]
:end-before: [END howto_operator_local_to_drive_upload_single_file]
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@
# under the License.
"""
Example DAG using LocalFilesystemToGoogleDriveOperator.
Using this operator requires the following additional scopes:
https://www.googleapis.com/auth/drive
"""
from __future__ import annotations

Expand All @@ -26,16 +29,27 @@
from airflow import models
from airflow.providers.google.suite.transfers.local_to_drive import LocalFilesystemToGoogleDriveOperator

SINGLE_FILE_LOCAL_PATHS = [Path("test1")]
MULTIPLE_FILES_LOCAL_PATHS = [Path("test1"), Path("test2")]
DRIVE_FOLDER = Path("test-folder")
DAG_ID = "example_local_to_drive"

FILE_NAME_1 = "test1"
FILE_NAME_2 = "test2"

LOCAL_PATH = str(Path(__file__).parent / "resources")

SINGLE_FILE_LOCAL_PATHS = [str(Path(LOCAL_PATH) / FILE_NAME_1)]
MULTIPLE_FILES_LOCAL_PATHS = [str(Path(LOCAL_PATH) / FILE_NAME_1), str(Path(LOCAL_PATH) / FILE_NAME_2)]

DRIVE_FOLDER = "test-folder"


with models.DAG(
"example_local_to_drive",
DAG_ID,
schedule="@once",
start_date=datetime(2021, 1, 1),
catchup=False,
tags=["example"],
) as dag:

# [START howto_operator_local_to_drive_upload_single_file]
upload_single_file = LocalFilesystemToGoogleDriveOperator(
task_id="upload_single_file",
Expand All @@ -53,4 +67,19 @@
)
# [END howto_operator_local_to_drive_upload_multiple_files]

upload_single_file >> upload_multiple_files
(
# TEST BODY
upload_single_file
>> upload_multiple_files
)

from tests.system.utils.watcher import watcher

# This test needs watcher in order to properly mark success/failure
# when "tearDown" task with trigger rule is part of the DAG
list(dag.tasks) >> watcher()

from tests.system.utils import get_test_run # noqa: E402

# Needed to run the example DAG with pytest (see: tests/system/README.md#run_via_pytest)
test_run = get_test_run(dag)
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
Expand Down
Empty file.
Empty file.

0 comments on commit b54a2de

Please sign in to comment.