feat: Use session temp tables for all ephemeral storage

[TrevorBergeron]

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

• Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
• Ensure the tests and linter pass
• Code coverage does not decrease (if any source code was changed)
• Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

[tswast]

FWIW: if you sync to main I made a similar fix in https://github.com/googleapis/python-bigquery-dataframes/pull/1572/files. I think we can keep the pytest.raises and just do the loop.

[TrevorBergeron]

ok, just merged in and used your version

[tswast]

Thanks! Just a few nits.

[tswast]

Nit: We can make this a little more internally consistent. If you'd like to use getattr in the if statement, let's use the := (walrus operator) so that we aren't fetching from self more than once.

Suggested change

	anon_dataset_manager = getattr(self, "_anon_dataset_manager", None)
	if anon_dataset_manager:
	self._anon_dataset_manager.close()
	if getattr(self, "_session_resource_manager", None):
	if self._session_resource_manager is not None:
	self._session_resource_manager.close()
	if anon_dataset_manager := getattr(self, "_anon_dataset_manager", None):
	anon_dataset_manager.close()
	if session_resource_manager := getattr(self, "_session_resource_manager", None):
	session_resource_manager.close()

[tswast]

Committed my own suggestion. 🤞 hopefully I didn't break anything.

[tswast]

Does this mean the table isn't created right away? If so, I think we might need to supply a session ID in the load job, if available.

Edit: I see this was moved to read_bigquery_load_job, which makes sense to me. Aside: with more hybrid engine stuff in the future, I can imagine some cases where to_gbq() would be doing a load job to a user-managed table, but I suppose that would probably use a very different job config, anyway.

[tswast]

I'm curious: why ALLOW_FIELD_ADDITION here, but using WRITE_TRUNCATE for read_gbq_load_job? Might be worthwhile to add some comments.

[TrevorBergeron]

I just want to make sure the ordering_col does not get overridden, as that is what is clustered.
Might still work with TRUNCATE as well?

[tswast]

I assume _storage_manager.create_temp_table handles this in the anonymous dataset case now?

Edit: Yes, I see we do set an expiration above:

        expiration = (
            datetime.datetime.now(datetime.timezone.utc) + constants.DEFAULT_EXPIRATION
        )
        table = bf_io_bigquery.create_temp_table(
            self.bqclient,
            self.allocate_temp_table(),
            expiration,
            schema=schema,
            cluster_columns=list(cluster_cols),
            kms_key=self._kms_key,
        )