Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: googleapis/python-bigquery-dataframes
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.11.1
Choose a base ref
...
head repository: googleapis/python-bigquery-dataframes
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v1.12.0
Choose a head ref

Commits on Jul 9, 2024

  1. chore(python): use python 3.10 for docs build (#828)

    Source-Link: googleapis/synthtool@9ae0785
    Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:52210e0e0559f5ea8c52be148b33504022e1faef4e95fbe4b32d68022af2fa7e
    
    Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
    Co-authored-by: Anthonios Partheniou <partheniou@google.com>
    3 people authored Jul 9, 2024
    Copy the full SHA
    f3b0f54 View commit details
  2. Copy the full SHA
    cdfd979 View commit details

Commits on Jul 10, 2024

  1. refactor: read transformer output columns from model entity (#817)

    * refactor: read transformer output columns from model entity
    
    * fix tests and docs
    
    * remove dup code
    
    * fix comment
    GarrettWu authored Jul 10, 2024
    Copy the full SHA
    eaa1db0 View commit details
  2. feat: add stratify param support to ml.model_selection.train_test_spl…

    …it method (#815)
    
    * feat: add stratify param to ml.model_selection.train_test_split
    
    * fix mypy
    
    * add notes for limit
    GarrettWu authored Jul 10, 2024
    Copy the full SHA
    27f8631 View commit details

Commits on Jul 11, 2024

  1. Copy the full SHA
    0d24f73 View commit details
  2. Copy the full SHA
    d7b333f View commit details
  3. Copy the full SHA
    8d1a03a View commit details
  4. Copy the full SHA
    4e7e67b View commit details

Commits on Jul 12, 2024

  1. Copy the full SHA
    93785cb View commit details
  2. Copy the full SHA
    c6d1c7c View commit details
  3. Copy the full SHA
    877bcc0 View commit details

Commits on Jul 15, 2024

  1. feat: support remote function cleanup with session.close (#818)

    * feat: support remote function cleanup with `session.close`
    
    * accept the possibility that the artifact may have already been deleted
    
    * add cleanup by previous session id
    
    * add more documentation
    
    * hold session artifacts in a remote function session class
    
    * fix the missing return keyword
    shobsi authored Jul 15, 2024
    Copy the full SHA
    ed06436 View commit details

Commits on Jul 16, 2024

  1. Copy the full SHA
    ab0dabc View commit details
  2. Copy the full SHA
    6278e0b View commit details
  3. Copy the full SHA
    c9eaff0 View commit details

Commits on Jul 18, 2024

  1. Copy the full SHA
    ff2faed View commit details
  2. refactor: Allow unambiguous windows even in unstrictly ordered sessio…

    …ns (#849)
    
    Co-authored-by: Huan Chen <142538604+Genesis929@users.noreply.github.com>
    TrevorBergeron and Genesis929 authored Jul 18, 2024
    Copy the full SHA
    83f254a View commit details
  3. test: re-enable gemini tuning load test (#846)

    * test: re-enable gemini tuning load test
    
    * remove commented out
    GarrettWu authored Jul 18, 2024
    Copy the full SHA
    827007c View commit details
  4. Copy the full SHA
    3346494 View commit details
  5. test: temporarily disable streaming tests (#850)

    * test: temporarily disable streaming tests
    
    * fix import
    shobsi authored Jul 18, 2024
    Copy the full SHA
    676a410 View commit details

Commits on Jul 19, 2024

  1. test: restore remote function stickiness in small tests (#847)

    * feat: support remote function cleanup with `session.close`
    
    * accept the possibility that the artifact may have already been deleted
    
    * add cleanup by previous session id
    
    * add more documentation
    
    * hold session artifacts in a remote function session class
    
    * fix the missing return keyword
    
    * test: restore stickiness in small `remote_function` tests
    
    docs: make `close_session`/`reset_session` appears in the docs
    shobsi authored Jul 19, 2024
    Copy the full SHA
    01d6bbb View commit details

Commits on Jul 23, 2024

  1. test: fix mypy failures to unblock presubmit tests (#852)

    The presubmit and continuous tests have started failing due to mypy. This change finds the failure points and puts in workarounds.
    
    Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
    - [ ] Make sure to open an issue as a [bug/issue](https://togithub.com/googleapis/python-bigquery-dataframes/issues/new/choose) before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
    - [ ] Ensure the tests and linter pass
    - [ ] Code coverage does not decrease (if any source code was changed)
    - [ ] Appropriate docs were updated (if necessary)
    
    Fixes #<issue_number_goes_here> 🦕
    shobsi authored Jul 23, 2024
    Copy the full SHA
    f9e4435 View commit details
  2. Copy the full SHA
    1b6a556 View commit details
  3. Copy the full SHA
    10da997 View commit details

Commits on Jul 25, 2024

  1. Copy the full SHA
    eb0ef75 View commit details
  2. Copy the full SHA
    823c0ce View commit details

Commits on Jul 26, 2024

  1. chore: increase stale cloud functions cleanup rate (#863)

    * chore: increase stale cloud functions cleanup rate
    
    * reword the comment
    shobsi authored Jul 26, 2024
    Copy the full SHA
    2c81086 View commit details

Commits on Jul 29, 2024

  1. chore(deps): update all dependencies (#715)

    * chore(deps): update all dependencies
    
    * 🦉 Updates from OwlBot post-processor
    
    See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md
    
    ---------
    
    Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
    Co-authored-by: Chelsea Lin <124939984+chelsea-lin@users.noreply.github.com>
    3 people authored Jul 29, 2024
    Copy the full SHA
    0676f73 View commit details
  2. Copy the full SHA
    b9e6150 View commit details

Commits on Jul 30, 2024

  1. chore(deps): update all dependencies (#866)

    * chore(deps): update all dependencies
    
    * revert pyarrow change
    
    Can't update pyarrow until we update ibis.
    
    ---------
    
    Co-authored-by: Tim Sweña (Swast) <swast@google.com>
    renovate-bot and tswast authored Jul 30, 2024
    Copy the full SHA
    f23de1a View commit details
  2. Copy the full SHA
    8e04c38 View commit details
  3. feat: Allow DataFrame.join for self-join on Null index (#860)

    * feat: Allow DataFrame.join for self-join on Null index
    
    * fix ml caching to apply post-join, add test
    
    * fix ml golden sql test
    
    * change unordered test to use linear regression
    TrevorBergeron authored Jul 30, 2024
    Copy the full SHA
    e950533 View commit details
  4. Copy the full SHA
    d0ab9cc View commit details
  5. fix: reduce redundant remote_function deployments (#856)

    * fix: reduce redundant `remote_function` deployments
    
    * do filename override in the naming rather than pickling
    
    * update documentation
    
    * update documentation
    shobsi authored Jul 30, 2024
    Copy the full SHA
    cbf2d42 View commit details

Commits on Jul 31, 2024

  1. feat: add streaming.StreamingDataFrame class (#864)

    * feat: add StreamingDataFrame support
    
    * use setattr for properties
    
    * fix bug
    
    * read session from DF
    
    * fix docs and tests
    
    * fix test
    
    * add preview warning
    
    * resolve comments
    
    * move to streaming.read_gbq_table, add logger
    
    * fix unit test
    
    * fix doc test
    
    * update notebook
    
    * add back preview warning
    GarrettWu authored Jul 31, 2024
    Copy the full SHA
    a7d7197 View commit details
  2. chore(main): release 1.12.0 (#834)

    Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
    release-please[bot] authored Jul 31, 2024
    Copy the full SHA
    8e00fe2 View commit details
Showing with 3,848 additions and 1,863 deletions.
  1. +2 −2 .github/.OwlBot.lock.yaml
  2. +12 −9 .kokoro/docker/docs/Dockerfile
  3. +22 −18 .kokoro/docker/docs/requirements.txt
  4. +23 −23 .kokoro/requirements.txt
  5. +27 −0 CHANGELOG.md
  6. +26 −7 bigframes/_config/bigquery_options.py
  7. +2 −0 bigframes/constants.py
  8. +9 −12 bigframes/core/__init__.py
  9. +105 −52 bigframes/core/blocks.py
  10. +2 −8 bigframes/core/compile/__init__.py
  11. +45 −32 bigframes/core/compile/api.py
  12. +28 −18 bigframes/core/compile/compiled.py
  13. +246 −241 bigframes/core/compile/compiler.py
  14. +2 −0 bigframes/core/groupby/__init__.py
  15. +1 −3 bigframes/core/indexes/base.py
  16. +9 −0 bigframes/core/join_def.py
  17. +37 −2 bigframes/core/nodes.py
  18. +68 −12 bigframes/core/ordering.py
  19. +38 −36 bigframes/core/rewrite.py
  20. +4 −0 bigframes/core/utils.py
  21. +8 −5 bigframes/core/validations.py
  22. +53 −35 bigframes/dataframe.py
  23. +605 −424 bigframes/functions/remote_function.py
  24. +9 −9 bigframes/functions/remote_function_template.py
  25. +18 −0 bigframes/ml/base.py
  26. +6 −14 bigframes/ml/compose.py
  27. +4 −4 bigframes/ml/core.py
  28. +13 −19 bigframes/ml/impute.py
  29. +38 −2 bigframes/ml/model_selection.py
  30. +1 −3 bigframes/ml/pipeline.py
  31. +86 −134 bigframes/ml/preprocessing.py
  32. +33 −8 bigframes/pandas/__init__.py
  33. +36 −18 bigframes/series.py
  34. +101 −29 bigframes/session/__init__.py
  35. +4 −0 bigframes/session/_io/bigquery/read_gbq_table.py
  36. +13 −243 bigframes/streaming/__init__.py
  37. +504 −0 bigframes/streaming/dataframe.py
  38. +1 −1 bigframes/version.py
  39. +104 −34 notebooks/dataframes/integrations.ipynb
  40. +1 −1 notebooks/regression/sklearn_linear_regression.ipynb
  41. +116 −116 notebooks/remote_functions/remote_function_usecases.ipynb
  42. +535 −0 notebooks/streaming/streaming_dataframe.ipynb
  43. +2 −3 noxfile.py
  44. +1 −1 samples/polars/requirements-test.txt
  45. +2 −2 samples/polars/requirements.txt
  46. +1 −1 samples/snippets/requirements-test.txt
  47. +1 −1 samples/snippets/requirements.txt
  48. +5 −1 scripts/create_bigtable.py
  49. +37 −6 tests/system/conftest.py
  50. +44 −0 tests/system/large/ml/test_linear_model.py
  51. +147 −3 tests/system/large/test_remote_function.py
  52. +4 −6 tests/system/large/test_session.py
  53. +18 −16 tests/system/large/test_streaming.py
  54. +24 −60 tests/system/load/test_llm.py
  55. +17 −24 tests/system/small/bigquery/test_json.py
  56. +1 −0 tests/system/small/ml/test_core.py
  57. +46 −6 tests/system/small/ml/test_llm.py
  58. +62 −0 tests/system/small/ml/test_model_selection.py
  59. +2 −0 tests/system/small/ml/test_remote.py
  60. +124 −44 tests/system/small/test_dataframe.py
  61. +14 −0 tests/system/small/test_null_index.py
  62. +131 −71 tests/system/small/test_remote_function.py
  63. +38 −0 tests/system/small/test_series.py
  64. +71 −0 tests/system/small/test_unordered.py
  65. +18 −0 tests/system/utils.py
  66. +1 −0 tests/unit/ml/test_golden_sql.py
  67. +2 −0 tests/unit/test_pandas.py
  68. +0 −15 third_party/bigframes_vendored/ibis/README.md
  69. +2 −8 third_party/bigframes_vendored/pandas/README.md
  70. +9 −5 third_party/bigframes_vendored/pandas/core/frame.py
  71. +26 −15 third_party/bigframes_vendored/pandas/core/generic.py
  72. +1 −1 third_party/bigframes_vendored/pandas/core/series.py
4 changes: 2 additions & 2 deletions .github/.OwlBot.lock.yaml
Original file line number Diff line number Diff line change
@@ -13,5 +13,5 @@
# limitations under the License.
docker:
image: gcr.io/cloud-devrel-public-resources/owlbot-python:latest
digest: sha256:d3de8a02819f65001effcbd3ea76ce97e9bcff035c7a89457f40f892c87c5b32
# created: 2024-07-03T17:43:00.77142528Z
digest: sha256:52210e0e0559f5ea8c52be148b33504022e1faef4e95fbe4b32d68022af2fa7e
# created: 2024-07-08T19:25:35.862283192Z
21 changes: 12 additions & 9 deletions .kokoro/docker/docs/Dockerfile
Original file line number Diff line number Diff line change
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

from ubuntu:22.04
from ubuntu:24.04

ENV DEBIAN_FRONTEND noninteractive

@@ -40,7 +40,6 @@ RUN apt-get update \
libssl-dev \
libsqlite3-dev \
portaudio19-dev \
python3-distutils \
redis-server \
software-properties-common \
ssh \
@@ -60,18 +59,22 @@ RUN apt-get update \
&& rm -rf /var/lib/apt/lists/* \
&& rm -f /var/cache/apt/archives/*.deb

###################### Install python 3.9.13

# Download python 3.9.13
RUN wget https://www.python.org/ftp/python/3.9.13/Python-3.9.13.tgz
###################### Install python 3.10.14 for docs/docfx session

# Download python 3.10.14
RUN wget https://www.python.org/ftp/python/3.10.14/Python-3.10.14.tgz

# Extract files
RUN tar -xvf Python-3.9.13.tgz
RUN tar -xvf Python-3.10.14.tgz

# Install python 3.9.13
RUN ./Python-3.9.13/configure --enable-optimizations
# Install python 3.10.14
RUN ./Python-3.10.14/configure --enable-optimizations
RUN make altinstall

RUN python3.10 -m venv /venv
ENV PATH /venv/bin:$PATH

###################### Install pip
RUN wget -O /tmp/get-pip.py 'https://bootstrap.pypa.io/get-pip.py' \
&& python3 /tmp/get-pip.py \
@@ -84,4 +87,4 @@ RUN python3 -m pip
COPY requirements.txt /requirements.txt
RUN python3 -m pip install --require-hashes -r requirements.txt

CMD ["python3.8"]
CMD ["python3.10"]
40 changes: 22 additions & 18 deletions .kokoro/docker/docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -4,9 +4,9 @@
#
# pip-compile --allow-unsafe --generate-hashes requirements.in
#
argcomplete==3.2.3 \
--hash=sha256:bf7900329262e481be5a15f56f19736b376df6f82ed27576fa893652c5de6c23 \
--hash=sha256:c12355e0494c76a2a7b73e3a59b09024ca0ba1e279fb9ed6c1b82d5b74b6a70c
argcomplete==3.4.0 \
--hash=sha256:69a79e083a716173e5532e0fa3bef45f793f4e61096cf52b5a42c0211c8b8aa5 \
--hash=sha256:c2abcdfe1be8ace47ba777d4fce319eb13bf8ad9dace8d085dcad6eded88057f
# via nox
colorlog==6.8.2 \
--hash=sha256:3e3e079a41feb5a1b64f978b5ea4f46040a94f11f0e8bbb8261e3dbbeca64d44 \
@@ -16,23 +16,27 @@ distlib==0.3.8 \
--hash=sha256:034db59a0b96f8ca18035f36290806a9a6e6bd9d1ff91e45a7f172eb17e51784 \
--hash=sha256:1530ea13e350031b6312d8580ddb6b27a104275a31106523b8f123787f494f64
# via virtualenv
filelock==3.13.1 \
--hash=sha256:521f5f56c50f8426f5e03ad3b281b490a87ef15bc6c526f168290f0c7148d44e \
--hash=sha256:57dbda9b35157b05fb3e58ee91448612eb674172fab98ee235ccb0b5bee19a1c
filelock==3.15.4 \
--hash=sha256:2207938cbc1844345cb01a5a95524dae30f0ce089eba5b00378295a17e3e90cb \
--hash=sha256:6ca1fffae96225dab4c6eaf1c4f4f28cd2568d3ec2a44e15a08520504de468e7
# via virtualenv
nox==2024.3.2 \
--hash=sha256:e53514173ac0b98dd47585096a55572fe504fecede58ced708979184d05440be \
--hash=sha256:f521ae08a15adbf5e11f16cb34e8d0e6ea521e0b92868f684e91677deb974553
nox==2024.4.15 \
--hash=sha256:6492236efa15a460ecb98e7b67562a28b70da006ab0be164e8821177577c0565 \
--hash=sha256:ecf6700199cdfa9e5ea0a41ff5e6ef4641d09508eda6edb89d9987864115817f
# via -r requirements.in
packaging==24.0 \
--hash=sha256:2ddfb553fdf02fb784c234c7ba6ccc288296ceabec964ad2eae3777778130bc5 \
--hash=sha256:eb82c5e3e56209074766e6885bb04b8c38a0c015d0a30036ebe7ece34c9989e9
packaging==24.1 \
--hash=sha256:026ed72c8ed3fcce5bf8950572258698927fd1dbda10a5e981cdf0ac37f4f002 \
--hash=sha256:5b8f2217dbdbd2f7f384c41c628544e6d52f2d0f53c6d0c3ea61aa5d1d7ff124
# via nox
platformdirs==4.2.0 \
--hash=sha256:0614df2a2f37e1a662acbd8e2b25b92ccf8632929bc6d43467e17fe89c75e068 \
--hash=sha256:ef0cc731df711022c174543cb70a9b5bd22e5a9337c8624ef2c2ceb8ddad8768
platformdirs==4.2.2 \
--hash=sha256:2d7a1657e36a80ea911db832a8a6ece5ee53d8de21edd5cc5879af6530b1bfee \
--hash=sha256:38b7b51f512eed9e84a22788b4bce1de17c0adb134d6becb09836e37d8654cd3
# via virtualenv
virtualenv==20.25.1 \
--hash=sha256:961c026ac520bac5f69acb8ea063e8a4f071bcc9457b9c1f28f6b085c511583a \
--hash=sha256:e08e13ecdca7a0bd53798f356d5831434afa5b07b93f0abdf0797b7a06ffe197
tomli==2.0.1 \
--hash=sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc \
--hash=sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f
# via nox
virtualenv==20.26.3 \
--hash=sha256:4c43a2a236279d9ea36a0d76f98d84bd6ca94ac4e0f4a3b9d46d05e10fea542a \
--hash=sha256:8cc4a31139e796e9a7de2cd5cf2489de1217193116a8fd42328f1bd65f434589
# via nox
46 changes: 23 additions & 23 deletions .kokoro/requirements.txt
Original file line number Diff line number Diff line change
@@ -20,9 +20,9 @@ cachetools==5.3.3 \
--hash=sha256:0abad1021d3f8325b2fc1d2e9c8b9c9d57b04c3932657a72465447332c24d945 \
--hash=sha256:ba29e2dfa0b8b556606f097407ed1aa62080ee108ab0dc5ec9d6a723a007d105
# via google-auth
certifi==2024.6.2 \
--hash=sha256:3cd43f1c6fa7dedc5899d69d3ad0398fd018ad1a17fba83ddaf78aa46c747516 \
--hash=sha256:ddc6c8ce995e6987e7faf5e3f1b02b302836a0e5d98ece18392cb1a36c72ad56
certifi==2024.7.4 \
--hash=sha256:5a1e7645bc0ec61a09e26c36f6106dd4cf40c6db3a1fb6352b0244e7fb057c7b \
--hash=sha256:c198e21b1289c2ab85ee4e67bb4b4ef3ead0892059901a8d5b622f24a1101e90
# via requests
cffi==1.16.0 \
--hash=sha256:0c9ef6ff37e974b73c25eecc13952c55bceed9112be2d9d938ded8e856138bcc \
@@ -371,23 +371,23 @@ more-itertools==10.3.0 \
# via
# jaraco-classes
# jaraco-functools
nh3==0.2.17 \
--hash=sha256:0316c25b76289cf23be6b66c77d3608a4fdf537b35426280032f432f14291b9a \
--hash=sha256:1a814dd7bba1cb0aba5bcb9bebcc88fd801b63e21e2450ae6c52d3b3336bc911 \
--hash=sha256:1aa52a7def528297f256de0844e8dd680ee279e79583c76d6fa73a978186ddfb \
--hash=sha256:22c26e20acbb253a5bdd33d432a326d18508a910e4dcf9a3316179860d53345a \
--hash=sha256:40015514022af31975c0b3bca4014634fa13cb5dc4dbcbc00570acc781316dcc \
--hash=sha256:40d0741a19c3d645e54efba71cb0d8c475b59135c1e3c580f879ad5514cbf028 \
--hash=sha256:551672fd71d06cd828e282abdb810d1be24e1abb7ae2543a8fa36a71c1006fe9 \
--hash=sha256:66f17d78826096291bd264f260213d2b3905e3c7fae6dfc5337d49429f1dc9f3 \
--hash=sha256:85cdbcca8ef10733bd31f931956f7fbb85145a4d11ab9e6742bbf44d88b7e351 \
--hash=sha256:a3f55fabe29164ba6026b5ad5c3151c314d136fd67415a17660b4aaddacf1b10 \
--hash=sha256:b4427ef0d2dfdec10b641ed0bdaf17957eb625b2ec0ea9329b3d28806c153d71 \
--hash=sha256:ba73a2f8d3a1b966e9cdba7b211779ad8a2561d2dba9674b8a19ed817923f65f \
--hash=sha256:c21bac1a7245cbd88c0b0e4a420221b7bfa838a2814ee5bb924e9c2f10a1120b \
--hash=sha256:c551eb2a3876e8ff2ac63dff1585236ed5dfec5ffd82216a7a174f7c5082a78a \
--hash=sha256:c790769152308421283679a142dbdb3d1c46c79c823008ecea8e8141db1a2062 \
--hash=sha256:d7a25fd8c86657f5d9d576268e3b3767c5cd4f42867c9383618be8517f0f022a
nh3==0.2.18 \
--hash=sha256:0411beb0589eacb6734f28d5497ca2ed379eafab8ad8c84b31bb5c34072b7164 \
--hash=sha256:14c5a72e9fe82aea5fe3072116ad4661af5cf8e8ff8fc5ad3450f123e4925e86 \
--hash=sha256:19aaba96e0f795bd0a6c56291495ff59364f4300d4a39b29a0abc9cb3774a84b \
--hash=sha256:34c03fa78e328c691f982b7c03d4423bdfd7da69cd707fe572f544cf74ac23ad \
--hash=sha256:36c95d4b70530b320b365659bb5034341316e6a9b30f0b25fa9c9eff4c27a204 \
--hash=sha256:3a157ab149e591bb638a55c8c6bcb8cdb559c8b12c13a8affaba6cedfe51713a \
--hash=sha256:42c64511469005058cd17cc1537578eac40ae9f7200bedcfd1fc1a05f4f8c200 \
--hash=sha256:5f36b271dae35c465ef5e9090e1fdaba4a60a56f0bb0ba03e0932a66f28b9189 \
--hash=sha256:6955369e4d9f48f41e3f238a9e60f9410645db7e07435e62c6a9ea6135a4907f \
--hash=sha256:7b7c2a3c9eb1a827d42539aa64091640bd275b81e097cd1d8d82ef91ffa2e811 \
--hash=sha256:8ce0f819d2f1933953fca255db2471ad58184a60508f03e6285e5114b6254844 \
--hash=sha256:94a166927e53972a9698af9542ace4e38b9de50c34352b962f4d9a7d4c927af4 \
--hash=sha256:a7f1b5b2c15866f2db413a3649a8fe4fd7b428ae58be2c0f6bca5eefd53ca2be \
--hash=sha256:c8b3a1cebcba9b3669ed1a84cc65bf005728d2f0bc1ed2a6594a992e817f3a50 \
--hash=sha256:de3ceed6e661954871d6cd78b410213bdcb136f79aafe22aa7182e028b8c7307 \
--hash=sha256:f0eca9ca8628dbb4e916ae2491d72957fdd35f7a5d326b7032a345f111ac07fe
# via readme-renderer
nox==2024.4.15 \
--hash=sha256:6492236efa15a460ecb98e7b67562a28b70da006ab0be164e8821177577c0565 \
@@ -460,9 +460,9 @@ python-dateutil==2.9.0.post0 \
--hash=sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3 \
--hash=sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427
# via gcp-releasetool
readme-renderer==43.0 \
--hash=sha256:1818dd28140813509eeed8d62687f7cd4f7bad90d4db586001c5dc09d4fde311 \
--hash=sha256:19db308d86ecd60e5affa3b2a98f017af384678c63c88e5d4556a380e674f3f9
readme-renderer==44.0 \
--hash=sha256:2fbca89b81a08526aadf1357a8c2ae889ec05fb03f5da67f9769c9a592166151 \
--hash=sha256:8712034eabbfa6805cacf1402b4eeb2a73028f72d1166d6f5cb7f9c047c5d1e1
# via twine
requests==2.32.3 \
--hash=sha256:55365417734eb18255590a9ff9eb97e9e1da868d4ccd6402399eaf68af20a760 \
27 changes: 27 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -4,6 +4,33 @@

[1]: https://pypi.org/project/bigframes/#history

## [1.12.0](https://github.com/googleapis/python-bigquery-dataframes/compare/v1.11.1...v1.12.0) (2024-07-31)


### Features

* Add bigframes-mode label to query jobs ([#832](https://github.com/googleapis/python-bigquery-dataframes/issues/832)) ([c9eaff0](https://github.com/googleapis/python-bigquery-dataframes/commit/c9eaff0a1a0731b28f4c67bca5606db12a47c8c0))
* Add config option to set partial ordering mode ([#855](https://github.com/googleapis/python-bigquery-dataframes/issues/855)) ([823c0ce](https://github.com/googleapis/python-bigquery-dataframes/commit/823c0ce57611c0918a9e9999638d7393337fe9af))
* Add stratify param support to ml.model_selection.train_test_split method ([#815](https://github.com/googleapis/python-bigquery-dataframes/issues/815)) ([27f8631](https://github.com/googleapis/python-bigquery-dataframes/commit/27f8631be81a3e136cfeb8904558bb4f3f5caa05))
* Add streaming.StreamingDataFrame class ([#864](https://github.com/googleapis/python-bigquery-dataframes/issues/864)) ([a7d7197](https://github.com/googleapis/python-bigquery-dataframes/commit/a7d7197a32c55b989ae4ea8f6cf6e1c0f7184cd4))
* Allow DataFrame.join for self-join on Null index ([#860](https://github.com/googleapis/python-bigquery-dataframes/issues/860)) ([e950533](https://github.com/googleapis/python-bigquery-dataframes/commit/e95053372c36ea5a91a2d7295c1a3a3671181670))
* Support remote function cleanup with `session.close` ([#818](https://github.com/googleapis/python-bigquery-dataframes/issues/818)) ([ed06436](https://github.com/googleapis/python-bigquery-dataframes/commit/ed06436612c0d46f190f79721416d473bde7e2f4))
* Support to_csv/parquet/json to local files/objects ([#858](https://github.com/googleapis/python-bigquery-dataframes/issues/858)) ([d0ab9cc](https://github.com/googleapis/python-bigquery-dataframes/commit/d0ab9cc47298bdde638299baecac9dffd7841ede))


### Bug Fixes

* Fewer relation joins from df self-operations ([#823](https://github.com/googleapis/python-bigquery-dataframes/issues/823)) ([0d24f73](https://github.com/googleapis/python-bigquery-dataframes/commit/0d24f737041c7dd70253ebb4baa8d8ef67bd4f1d))
* Fix 'sql' property for null index ([#844](https://github.com/googleapis/python-bigquery-dataframes/issues/844)) ([1b6a556](https://github.com/googleapis/python-bigquery-dataframes/commit/1b6a556206a7a66283339d827ab12db2753521e2))
* Fix unordered mode using ordered path to print frame ([#839](https://github.com/googleapis/python-bigquery-dataframes/issues/839)) ([93785cb](https://github.com/googleapis/python-bigquery-dataframes/commit/93785cb48be4a2eb8770129148bd0b897fed4ee7))
* Reduce redundant `remote_function` deployments ([#856](https://github.com/googleapis/python-bigquery-dataframes/issues/856)) ([cbf2d42](https://github.com/googleapis/python-bigquery-dataframes/commit/cbf2d42e4d961a7537381a9c3b28a8b463ad8f74))


### Documentation

* Add partner attribution steps to integrations sample notebook ([#835](https://github.com/googleapis/python-bigquery-dataframes/issues/835)) ([d7b333f](https://github.com/googleapis/python-bigquery-dataframes/commit/d7b333fa26acddaeb5ccca4f81b1d624dff03ba2))
* Make `get_global_session`/`close_session`/`reset_session` appears in the docs ([#847](https://github.com/googleapis/python-bigquery-dataframes/issues/847)) ([01d6bbb](https://github.com/googleapis/python-bigquery-dataframes/commit/01d6bbb7479da706dc62bb5e7d51dc28a4042812))

## [1.11.1](https://github.com/googleapis/python-bigquery-dataframes/compare/v1.11.0...v1.11.1) (2024-07-08)


33 changes: 26 additions & 7 deletions bigframes/_config/bigquery_options.py
Original file line number Diff line number Diff line change
@@ -16,7 +16,8 @@

from __future__ import annotations

from typing import Optional
from enum import Enum
from typing import Literal, Optional
import warnings

import google.api_core.exceptions
@@ -26,6 +27,12 @@
import bigframes.constants
import bigframes.exceptions


class OrderingMode(Enum):
STRICT = "strict"
PARTIAL = "partial"


SESSION_STARTED_MESSAGE = (
"Cannot change '{attribute}' once a session has started. "
"Call bigframes.pandas.close_session() first, if you are using the bigframes.pandas API."
@@ -57,6 +64,14 @@ def _validate_location(value: Optional[str]):
)


def _validate_ordering_mode(value: str) -> OrderingMode:
if value.casefold() == OrderingMode.STRICT.value.casefold():
return OrderingMode.STRICT
if value.casefold() == OrderingMode.PARTIAL.value.casefold():
return OrderingMode.PARTIAL
raise ValueError("Ordering mode must be one of 'strict' or 'partial'.")


class BigQueryOptions:
"""Encapsulates configuration for working with a session."""

@@ -71,7 +86,7 @@ def __init__(
kms_key_name: Optional[str] = None,
skip_bq_connection_check: bool = False,
*,
_strictly_ordered: bool = True,
ordering_mode: Literal["strict", "partial"] = "strict",
):
self._credentials = credentials
self._project = project
@@ -82,8 +97,8 @@ def __init__(
self._kms_key_name = kms_key_name
self._skip_bq_connection_check = skip_bq_connection_check
self._session_started = False
# Determines the ordering strictness for the session. For internal use only.
self._strictly_ordered_internal = _strictly_ordered
# Determines the ordering strictness for the session.
self._ordering_mode = _validate_ordering_mode(ordering_mode)

@property
def application_name(self) -> Optional[str]:
@@ -241,6 +256,10 @@ def kms_key_name(self, value: str):
self._kms_key_name = value

@property
def _strictly_ordered(self) -> bool:
"""Internal use only. Controls whether total row order is always maintained for DataFrame/Series."""
return self._strictly_ordered_internal
def ordering_mode(self) -> Literal["strict", "partial"]:
"""Controls whether total row order is always maintained for DataFrame/Series."""
return self._ordering_mode.value

@ordering_mode.setter
def ordering_mode(self, ordering_mode: Literal["strict", "partial"]) -> None:
self._ordering_mode = _validate_ordering_mode(ordering_mode)
2 changes: 2 additions & 0 deletions bigframes/constants.py
Original file line number Diff line number Diff line change
@@ -99,3 +99,5 @@

# BigQuery default is 10000, leave 100 for overhead
MAX_COLUMNS = 9900

SUGGEST_PEEK_PREVIEW = "Use .peek(n) to preview n arbitrary rows."
21 changes: 9 additions & 12 deletions bigframes/core/__init__.py
Original file line number Diff line number Diff line change
@@ -147,7 +147,7 @@ def _compiled_schema(self) -> schemata.ArraySchema:
def as_cached(
self: ArrayValue,
cache_table: google.cloud.bigquery.Table,
ordering: Optional[orderings.TotalOrdering],
ordering: Optional[orderings.RowOrdering],
) -> ArrayValue:
"""
Replace the node with an equivalent one that references a tabel where the value has been materialized to.
@@ -194,7 +194,7 @@ def promote_offsets(self, col_id: str) -> ArrayValue:
"""
Convenience function to promote copy of column offsets to a value column. Can be used to reset index.
"""
if not self.session._strictly_ordered:
if self.node.order_ambiguous and not self.session._strictly_ordered:
raise ValueError("Generating offsets not supported in unordered mode")
return ArrayValue(nodes.PromoteOffsetsNode(child=self.node, col_id=col_id))

@@ -346,7 +346,7 @@ def project_window_op(
"""
# TODO: Support non-deterministic windowing
if window_spec.row_bounded or not op.order_independent:
if not self.session._strictly_ordered:
if self.node.order_ambiguous and not self.session._strictly_ordered:
raise ValueError(
"Order-dependent windowed ops not supported in unordered mode"
)
@@ -460,9 +460,9 @@ def _cross_join_w_labels(
conditions=(), mappings=(*labels_mappings, *table_mappings), type="cross"
)
if join_side == "left":
joined_array = self.join(labels_array, join_def=join)
joined_array = self.relational_join(labels_array, join_def=join)
else:
joined_array = labels_array.join(self, join_def=join)
joined_array = labels_array.relational_join(self, join_def=join)
return joined_array

def _create_unpivot_labels_array(
@@ -485,30 +485,27 @@ def _create_unpivot_labels_array(

return ArrayValue.from_pyarrow(pa.Table.from_pylist(rows), session=self.session)

def join(
def relational_join(
self,
other: ArrayValue,
join_def: join_def.JoinDefinition,
allow_row_identity_join: bool = False,
):
) -> ArrayValue:
join_node = nodes.JoinNode(
left_child=self.node,
right_child=other.node,
join=join_def,
allow_row_identity_join=allow_row_identity_join,
)
if allow_row_identity_join:
return ArrayValue(bigframes.core.rewrite.maybe_rewrite_join(join_node))
return ArrayValue(join_node)

def try_align_as_projection(
self,
other: ArrayValue,
join_type: join_def.JoinType,
join_keys: typing.Tuple[join_def.CoalescedColumnMapping, ...],
mappings: typing.Tuple[join_def.JoinColumnMapping, ...],
) -> typing.Optional[ArrayValue]:
result = bigframes.core.rewrite.join_as_projection(
self.node, other.node, mappings, join_type
self.node, other.node, join_keys, mappings, join_type
)
if result is not None:
return ArrayValue(result)
Loading