Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Passing Custom Image Family Name to the DataprocClusterCreateOperator() #15245

Closed
pateash opened this issue Apr 7, 2021 · 3 comments · Fixed by #15250
Closed

Passing Custom Image Family Name to the DataprocClusterCreateOperator() #15245

pateash opened this issue Apr 7, 2021 · 3 comments · Fixed by #15250
Labels
kind:feature Feature Requests

Comments

@pateash
Copy link
Contributor

pateash commented Apr 7, 2021 •
edited
Loading

Description
Currently, we can only pass custom Image name to DataprocClusterCreateOperator(),
as the custom image expires after 60 days, we either need to update the image or we need to pass the expiration token.

Functionality is already available in gcloud and REST.

gcloud dataproc clusters test_cluster ...... --image projects/<custom_image_project_id>/global/images/family/<family_name> .......

Use case / motivation
The user should be able to pass either Custom Image or Custom Image family name,
so we don't have to update the image up on expiration or use expiration token.

Are you willing to submit a PR?
Yes

Related Issues
None
@pateash pateash added the kind:feature Feature Requests label Apr 7, 2021
@boring-cyborg
Copy link

boring-cyborg bot commented Apr 7, 2021

Thanks for opening your first issue here! Be sure to follow the issue template!

@pateash
Copy link
Contributor Author

pateash commented Apr 7, 2021 •
edited
Loading

I will work on this, please let me know any suggestions or comments.

@pateash
Copy link
Contributor Author

pateash commented Apr 7, 2021 •
edited
Loading

There is workaround to this, we can just pass family/<family_name> in custom_image parameter to DataprocClusterCreateOperator(), taking advantage of because the way custom_image_url is created.
we will get the exact URL for custom_image_family_name.

image

But I really think there should be a dedicated parameter custom_image_family for avoiding any confusion, and this substitution might not work in the future.

@pateash pateash mentioned this issue Apr 7, 2021
Merged
xinbinhuang added a commit that referenced this issue Apr 18, 2021
@pateash @xinbinhuang
[Airflow-15245] - passing custom image family name to the DataProcClu…
6da36ba 
…sterCreateoperator (#15250)

* [airflow-15245] - custom_image_family added as a parameter to DataprocCreateClusterOperator

Signed-off-by: ashish <[email protected]>

* [airflow-15245] - test added to check both custom_image and custom_image_family must not be passed

Signed-off-by: ashish <[email protected]>

* [airflow-#15245] - typo fixed in documentation

Signed-off-by: ashish <[email protected]>

* [Airflow-15245] - comments updated, more info provided.

* [Airflow-15245] - sanity check added for image_version and custom_image_family.

* Update airflow/providers/google/cloud/operators/dataproc.py

Co-authored-by: Xinbin Huang <[email protected]>

* Apply suggestions from code review

Co-authored-by: Xinbin Huang <[email protected]>

* [Airflow-15245] - added a test case to verify the generated cluster config is as expected with custom_image_family and single_node.

* Remove print() from test case

Co-authored-by: Ashish Patel <[email protected]>
Co-authored-by: Xinbin Huang <[email protected]>
@pateash pateash mentioned this issue May 3, 2021
Closed
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind:feature Feature Requests
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#15245 - passing custom image family name to the DataProcClusterCreateoperator
1 participant
@pateash