Skip to content

CHANGELOG.md

Latest commit

 

History

History
445 lines (310 loc) · 24.8 KB

CHANGELOG.md

File metadata and controls

445 lines (310 loc) · 24.8 KB
Raw

Release History

Unreleased

  • Kernel backend (use_kernel=True): OAuth U2M with auth_type="databricks-oauth" now forwards the connector's databricks-sql-python OAuth-app bundle (client_id + sql offline_access scopes + redirect port) into the kernel, so a bare U2M connection authenticates as databricks-sql-python — parity with the Thrift path — instead of inheriting the kernel's own databricks-sql-connector default. A caller-supplied oauth_client_id (with its coupled oauth_redirect_port) is honored, as is a caller-supplied oauth_scopes; absent one, the connector default (sql offline_access) is forwarded. Note: the kernel binds a single U2M redirect port, so unlike the Thrift path (which tries the full 8020..8024 range) the kernel path uses only one port and does not fall back to the next port if it is already bound — pass oauth_redirect_port (with oauth_client_id) to pick a free one on a port collision. auth_type="azure-oauth" (Azure AD) is not yet supported on the kernel path and raises NotSupportedError — use the Thrift backend for it (PECOBLR-4040; Azure tracked by PECOBLR-4120)

4.4.0 (2026-07-22)

  • Raised the minimum supported Python version to 3.10, dropping the end-of-life 3.8/3.9, to update the lockfile and clear CVE-flagged dependencies in the repo (#798)
  • Fix: REMOVE staging operations no longer require staging_allowed_local_path to be set, since removing a remote file does not touch the local filesystem (#726)
  • Report cursor.rowcount for DML on the Thrift backend: INSERT/UPDATE/DELETE/MERGE now set rowcount to the server's affected-row count instead of the hardcoded -1; SELECT (and statements the server does not report a count for) still return -1. executemany aggregates the count across all parameter sets per PEP 249 (#784)

4.3.0 (2026-06-12)

  • New: optional Rust kernel backend (use_kernel=True). Adds an alternative connection path backed by the native databricks-sql-kernel client (a Rust core exposed via PyO3), installable with the new databricks-sql-connector[kernel] extra. The kernel talks to Databricks over the SEA (Statement Execution API) HTTP transport — not Thrift — with CloudFetch and inline-Arrow result fetching, so use_kernel=True gives you a modern SEA-native client through the same DB-API surface. Supports PAT, OAuth M2M, and OAuth U2M auth. Requires Python >= 3.10 (the kernel wheel is cp310-abi3); on older interpreters the extra is a no-op and use_kernel=True raises a clear ImportError. The default backend remains Thrift — opt in per connection.
  • Kernel backend behavior is aligned with the Thrift backend so application code works the same either way: consistent cursor-state tracking (query_id / get_query_state), metadata (catalogs/schemas/tables/columns with JDBC-style filter semantics and case-insensitive table_types), DML rowcount, server-sourced async execution state, sync cancel(), fail-loud staging/volume operations, and structured error context (SQLSTATE, diagnostic info). Kernel logs surface through Python logging under the databricks.sql.kernel logger (#824, #825, #830, #838, #839 by @vikrantpuppala)
  • Revert the thrift 0.23.0 bump that broke installation on DBR LTS (ES-1960554) (#840 by @vikrantpuppala)

4.2.7 (2026-06-02)

  • Extract SPOG org-id from cluster http_path for non-Thrift requests (#817 by @msrathore-db)
  • Remove empty chunks in CloudFetch concatenation (#814 by @jprakash-db)
  • Add _retry_server_directed_only mode for Retry-After header compliance (#756 by @sd-db)
  • Bump thrift to 0.23.0 (#796 by @leoromanovsky)
  • Allow pandas 3.x in dependency constraints (#768 by @moomindani)
  • Telemetry: unwrap TokenFederationProvider to report inner auth mechanism/flow (#781 by @samikshya-db)

4.2.6 (2026-04-22)

  • Add SPOG routing support for account-level vanity URLs (#767 by @msrathore-db)
  • Fix dependency_manager: handle PEP 440 ~= compatible release syntax (#776 by @vikrantpuppala)
  • Bump thrift to fix deprecation warning (#733 by @Korijn)
  • Add AI coding agent detection to User-Agent header (#740 by @vikrantpuppala)
  • Add statement-level query_tags support for SEA backend (#754 by @sreekanth-db)
  • Update PyArrow concatenation of tables to use promote_options as default (#751 by @jprakash-db)
  • Fix float inference to use DoubleParameter (64-bit) instead of FloatParameter (#742 by @Shubhambhusate)
  • Allow specifying query_tags as a dict upon connection creation (#749 by @jiabin-hu)
  • Add query_tags parameter support for execute methods (#736 by @jiabin-hu)

4.2.5 (2026-02-09)

  • Fix feature-flag endpoint retries in gov region (#735 by @samikshya-db)
  • Improve telemetry lifecycle management (#734 by @msrathore-db)

4.2.4 (2026-01-07)

  • Fixed the exception handler close() on _TelemetryClientHolder (#723 by @msrathore-db)
  • Created util method to normalise http protocol in http path (#724 by @nikhilsuri-db)

4.2.3 (2025-12-18)

  • added pandas < 2.4.0 support and tests for py 3.14 (#720 by @sreekanth-db)
  • pandas 2.3.3 support for py < 3.14 (#721 by @sreekanth-db)

4.2.2 (2025-12-01)

  • Change default use_hybrid_disposition to False (#714 by @samikshya-db)
  • Circuit breaker changes using pybreaker (#705 by @nikhilsuri-db)
  • perf: Optimize telemetry latency logging to reduce overhead (#715 by @samikshya-db)
  • basic e2e test for force telemetry verification (#708 by @nikhilsuri-db)
  • Telemetry is ON by default to track connection stats. (Note : This strictly excludes PII, query text, and results) (#717 by @samikshya-db)

4.2.1 (2025-11-20)

  • Ignore transactions by default (#711 by @jayantsing-db)

4.2.0 (2025-11-14)

  • Add multi-statement transaction support (#704 by @jayantsing-db)
  • Add a workflow to parallelise the E2E tests (#697 by @msrathore-db)
  • Bring Python telemetry event model consistent with JDBC (#701 by @nikhilsuri-db)

4.1.4 (2025-10-15)

  • Add support for Token Federation (#691 by @madhav-db)
  • Add metric view support (#688 by @shivam2680)
  • Increased time limit for long running queries (#686 by @jprakash-db)

4.1.3 (2025-09-17)

  • Query tags integration (#663 by @sreekanth-db)
  • Add variant support (#560 by @shivam2680)

4.1.2 (2025-08-22)

  • Streaming ingestion support for PUT operation (#643 by @sreekanth-db)
  • Removed use_threads argument on concat_tables for compatibility with pyarrow<14 (#684 by @jprakash-db)

4.1.1 (2025-08-21)

  • Add documentation for proxy support (#680 by @vikrantpuppala)
  • Fix compatibility with urllib3<2 and add CI actions to improve dependency checks (#678 by @vikrantpuppala)

4.1.0 (2025-08-18)

  • Removed Codeowners (#623 by @jprakash-db)
  • Azure Service Principal Credential Provider (#621 by @jprakash-db)
  • Add optional telemetry support to the python connector (#628 by @saishreeeee)
  • Fix potential resource leak in CloudFetchQueue (#624 by @varun-edachali-dbx)
  • Generalise Backend Layer (#604 by @varun-edachali-dbx)
  • Arrow performance optimizations (#638 by @jprakash-db)
  • Connection errors to unauthenticated telemetry endpoint (#619 by @saishreeeee)
  • SEA: Execution Phase (#645 by @varun-edachali-dbx)
  • Add retry mechanism to telemetry requests (#617 by @saishreeeee)
  • SEA: Fetch Phase (#650 by @varun-edachali-dbx)
  • added logs for cloud fetch speed (#654 by @shivam2680)
  • Make telemetry batch size configurable and add time-based flush (#622 by @saishreeeee)
  • Normalise type code (#652 by @varun-edachali-dbx)
  • Testing for telemetry (#616 by @saishreeeee)
  • Bug fixes in telemetry (#659 by @saishreeeee)
  • Telemetry server-side flag integration (#646 by @saishreeeee)
  • Enhance SEA HTTP Client (#618 by @varun-edachali-dbx)
  • SEA: Allow large metadata responses (#653 by @varun-edachali-dbx)
  • Added code coverage workflow to test the code coverage from unit and e2e tests (#657 by @msrathore-db)
  • Concat tables to be backward compatible (#647 by @jprakash-db)
  • Refactor codebase to use a unified http client (#673 by @vikrantpuppala)
  • Add kerberos support for proxy auth (#675 by @vikrantpuppala)

4.0.5 (2025-06-24)

  • Fix: Reverted change in cursor close handling which led to errors impacting users (#613 by @madhav-db)

4.0.4 (2025-06-16)

  • Update thrift client library after cleaning up unused fields and structs (#553 by @vikrantpuppala)
  • Refactor decimal conversion in PyArrow tables to use direct casting (#544 by @jayantsing-db)
  • Fix: fetchall_arrow to always return results in arrow format (#551 by @shivam2680)
  • Enhance cursor close handling and context manager exception management to prevent server side resource leaks (#554 by @madhav-db)
  • Added additional logging to enhance debugging (#556 by @saishreeeee)
  • Feature: Added support for complex data types such as Arrays and Map [Private Preview] (#559 by @jprakash-db)

4.0.3 (2025-04-22)

  • Fix: Removed packaging dependency in favour of default libraries, for urllib3 version checks (#547 by @jprakash-db)

4.0.2 (2025-04-01)

  • Fix: Relaxed the pin for python-dateutil to be ^2.8.0 (#538 by @jprakash-db)

4.0.1 (2025-03-19)

Note: this release was yanked from Pypi on 19 March 2025 due to compatibility issues with dbt-databricks<1.5.3

  • Support for multiple timestamp formats parsing (#533 by @jprakash-db)
  • Rename _user_agent_entry in connect call to user_agent_entry to expose it as a public parameter. (#530 by @shivam2680)
  • Fix: compatibility with urllib3 versions less than 2.x. (#526 by @shivam2680)
  • Support for Python 3.13 and updated dependencies (#510 by @dhirschfeld and @dbaxa)

4.0.0 (2025-01-19)

  • Split the connector into two separate packages: databricks-sql-connector and databricks-sqlalchemy. The databricks-sql-connector package contains the core functionality of the connector, while the databricks-sqlalchemy package contains the SQLAlchemy dialect for the connector.
  • Pyarrow dependency is now optional in databricks-sql-connector. Users needing arrow are supposed to explicitly install pyarrow

3.7.3 (2025-03-28)

  • Fix: Unable to poll small results in execute_async function (#515 by @jprakash-db)
  • Updated log messages to show the status code and error messages of requests (#511 by @jprakash-db)
  • Fix: Incorrect metadata was fetched in case of queries with the same alias (#505 by @jprakash-db)

3.7.2 (2025-01-31)

  • Updated the retry_dela_max and retry_timeout (#497 by @jprakash-db)

3.7.1 (2025-01-07)

  • Relaxed the number of Http retry attempts (#486 by @jprakash-db)

3.7.0 (2024-12-23)

  • Fix: Incorrect number of rows fetched in inline results when fetching results with FETCH_NEXT orientation (#479 by @jprakash-db)
  • Updated the doc to specify native parameters are not supported in PUT operation (#477 by @jprakash-db)
  • Relax pyarrow and numpy pin (#452 by @arredond)
  • Feature: Support for async execute has been added (#463 by @jprakash-db)
  • Updated the HTTP retry logic to be similar to the other Databricks drivers (#467 by @jprakash-db)

3.6.0 (2024-10-25)

  • Support encryption headers in the cloud fetch request (#460 by @jackyhu-db)

3.5.0 (2024-10-18)

  • Create a non pyarrow flow to handle small results for the column set (#440 by @jprakash-db)
  • Fix: On non-retryable error, ensure PySQL includes useful information in error (#447 by @shivam2680)

3.4.0 (2024-08-27)

  • Unpin pandas to support v2.2.2 (#416 by @kfollesdal)
  • Make OAuth as the default authenticator if no authentication setting is provided (#419 by @jackyhu-db)
  • Fix (regression): use SSL options with HTTPS connection pool (#425 by @kravets-levko)

3.3.0 (2024-07-18)

  • Don't retry requests that fail with HTTP code 401 (#408 by @Hodnebo)
  • Remove username/password (aka "basic") auth option (#409 by @jackyhu-db)
  • Refactor CloudFetch handler to fix numerous issues with it (#405 by @kravets-levko)
  • Add option to disable SSL verification for CloudFetch links (#414 by @kravets-levko)

Databricks-managed passwords reached end of life on July 10, 2024. Therefore, Basic auth support was removed from the library. See https://docs.databricks.com/en/security/auth-authz/password-deprecation.html

The existing option _tls_no_verify=True of sql.connect(...) will now also disable SSL cert verification (but not the SSL itself) for CloudFetch links. This option should be used as a workaround only, when other ways to fix SSL certificate errors didn't work.

3.2.0 (2024-06-06)

  • Update proxy authentication (#354 by @amir-haroun)
  • Relax pyarrow pin (#389 by @dhirschfeld)
  • Fix error logging in OAuth manager (#269 by @susodapop)
  • SQLAlchemy: enable delta.feature.allowColumnDefaults for all tables (#343 by @dhirschfeld)
  • Update thrift dependency (#397 by @m1n0)

3.1.2 (2024-04-18)

  • Remove broken cookie code (#379)
  • Small typing fixes (#382, #384 thanks @wyattscarpenter)

3.1.1 (2024-03-19)

  • Don't retry requests that fail with code 403 (#373)
  • Assume a default retry-after for 429/503 (#371)
  • Fix boolean literals (#357)

3.1.0 (2024-02-16)

  • Revert retry-after behavior to be exponential backoff (#349)
  • Support Databricks OAuth on Azure (#351)
  • Support Databricks OAuth on GCP (#338)

3.0.3 (2024-02-02)

  • Revised docstrings and examples for OAuth (#339)
  • Redact the URL query parameters from the urllib3.connectionpool logs (#341)

3.0.2 (2024-01-25)

  • SQLAlchemy dialect now supports table and column comments (thanks @cbornet!)
  • Fix: SQLAlchemy dialect now correctly reflects TINYINT types (thanks @TimTheinAtTabs!)
  • Fix: server_hostname URIs that included https:// would raise an exception
  • Other: pinned to pandas<=2.1 and urllib3>=1.26 to avoid runtime errors in dbt-databricks (#330)

3.0.1 (2023-12-01)

  • Other: updated docstring comment about default parameterization approach (#287)
  • Other: added tests for reading complex types and revised docstrings and type hints (#293)
  • Fix: SQLAlchemy dialect raised DeprecationWarning due to dbapi classmethod (#294)
  • Fix: SQLAlchemy dialect could not reflect TIMESTAMP_NTZ columns (#296)

3.0.0 (2023-11-17)

  • Remove support for Python 3.7
  • Add support for native parameterized SQL queries. Requires DBR 14.2 and above. See docs/parameters.md for more info.
  • Completely rewritten SQLAlchemy dialect
    • Adds support for SQLAlchemy >= 2.0 and drops support for SQLAlchemy 1.x
    • Full e2e test coverage of all supported features
    • Detailed usage notes in README.sqlalchemy.md
    • Adds support for:
      • New types: TIME, TIMESTAMP, TIMESTAMP_NTZ, TINYINT
      • Numeric type scale and precision, like Numeric(10,2)
      • Reading and writing PrimaryKeyConstraint and ForeignKeyConstraint
      • Reading and writing composite keys
      • Reading and writing from views
      • Writing Identity to tables (i.e. autoincrementing primary keys)
      • LIMIT and OFFSET for paging through results
      • Caching metadata calls
  • Enable cloud fetch by default. To disable, set use_cloud_fetch=False when building databricks.sql.client.
  • Add integration tests for Databricks UC Volumes ingestion queries
  • Retries:
    • Add _retry_max_redirects config
    • Set _enable_v3_retries=True and warn if users override it
  • Security: bump minimum pyarrow version to 14.0.1 (CVE-2023-47248)

2.9.3 (2023-08-24)

  • Fix: Connections failed when urllib3~=1.0.0 is installed (#206)

2.9.2 (2023-08-17)

Note: this release was yanked from Pypi on 13 September 2023 due to compatibility issues with environments where urllib3<=2.0.0 were installed. The log changes are incorporated into version 2.9.3 and greater.

  • Other: Add examples/v3_retries_query_execute.py (#199)
  • Other: suppress log message when _enable_v3_retries is not True (#199)
  • Other: make this connector backwards compatible with urllib3>=1.0.0 (#197)

2.9.1 (2023-08-11)

Note: this release was yanked from Pypi on 13 September 2023 due to compatibility issues with environments where urllib3<=2.0.0 were installed.

  • Other: Explicitly pin urllib3 to ^2.0.0 (#191)

2.9.0 (2023-08-10)

  • Replace retry handling with DatabricksRetryPolicy. This is disabled by default. To enable, set _enable_v3_retries=True when creating databricks.sql.client (#182)
  • Other: Fix typo in README quick start example (#186)
  • Other: Add autospec to Client mocks and tidy up make_request (#188)

2.8.0 (2023-07-21)

  • Add support for Cloud Fetch. Disabled by default. Set use_cloud_fetch=True when building databricks.sql.client to enable it (#146, #151, #154)
  • SQLAlchemy has_table function now honours schema= argument and adds catalog= argument (#174)
  • SQLAlchemy set non_native_boolean_check_constraint False as it's not supported by Databricks (#120)
  • Fix: Revised SQLAlchemy dialect and examples for compatibility with SQLAlchemy==1.3.x (#173)
  • Fix: oauth would fail if expired credentials appeared in ~/.netrc (#122)
  • Fix: Python HTTP proxies were broken after switch to urllib3 (#158)
  • Other: remove unused import in SQLAlchemy dialect
  • Other: Relax pandas dependency constraint to allow ^2.0.0 (#164)
  • Other: Connector now logs operation handle guids as hexadecimal instead of bytes (#170)
  • Other: test_socket_timeout_user_defined e2e test was broken (#144)

2.7.0 (2023-06-26)

  • Fix: connector raised exception when calling close() on a closed Thrift session
  • Improve e2e test development ergonomics
  • Redact logged thrift responses by default
  • Add support for OAuth on Databricks Azure

2.6.2 (2023-06-14)

  • Fix: Retry GetOperationStatus requests for http errors

2.6.1 (2023-06-08)

  • Fix: http.client would raise a BadStatusLine exception in some cases

2.6.0 (2023-06-07)

  • Add support for HTTP 1.1 connections (connection pools)
  • Add a default socket timeout for thrift RPCs

2.5.2 (2023-05-08)

  • Fix: SQLAlchemy adapter could not reflect TIMESTAMP or DATETIME columns
  • Other: Relax pandas and alembic dependency specifications

2.5.1 (2023-04-28)

  • Other: Relax sqlalchemy required version as it was unecessarily strict.

2.5.0 (2023-04-14)

  • Add support for External Auth providers
  • Fix: Python HTTP proxies were broken
  • Other: All Thrift requests that timeout during connection will be automatically retried

2.4.1 (2023-03-21)

  • Less strict numpy and pyarrow dependencies
  • Update examples in README to use security best practices
  • Update docstring for client.execute() for clarity

2.4.0 (2023-02-21)

  • Improve compatibility when installed alongside other Databricks namespace Python packages
  • Add SQLAlchemy dialect

2.3.0 (2023-01-10)

  • Support staging ingestion commands for DBR 12+

2.2.2 (2023-01-03)

  • Support custom oauth client id and redirect port
  • Fix: Add none check on _oauth_persistence in DatabricksOAuthProvider

2.2.1 (2022-11-29)

  • Add support for Python 3.11

2.2.0 (2022-11-15)

2.1.0 (2022-09-30)

  • Introduce experimental OAuth support while Bring Your Own IDP is in Public Preview on AWS
  • Add functional examples

2.0.5 (2022-08-23)

  • Fix: closing a connection now closes any open cursors from that connection at the server
  • Other: Add project links to pyproject.toml (helpful for visitors from PyPi)

2.0.4 (2022-08-17)

  • Add support for Python 3.10
  • Add unit test matrix for supported Python versions

Huge thanks to @dbaxa for contributing this change!

2.0.3 (2022-08-05)

  • Add retry logic for GetOperationStatus requests that fail with an OSError
  • Reorganised code to use Poetry for dependency management.

2.0.2 (2022-05-04)

  • Better exception handling in automatic connection close

2.0.1 (2022-04-21)

  • Fixed Pandas dependency in setup.cfg to be >= 1.2.0

2.0.0 (2022-04-19)

  • Initial stable release of V2
  • Added better support for complex types, so that in Databricks runtime 10.3+, Arrays, Maps and Structs will get deserialized as lists, lists of tuples and dicts, respectively.
  • Changed the name of the metadata arg to http_headers

2.0.b2 (2022-04-04)

  • Change import of collections.Iterable to collections.abc.Iterable to make the library compatible with Python 3.10
  • Fixed bug with .tables method so that .tables works as expected with Unity-Catalog enabled endpoints

2.0.0b1 (2022-03-04)

  • Fix packaging issue (dependencies were not being installed properly)
  • Fetching timestamp results will now return aware instead of naive timestamps
  • The client will now default to using simplified error messages

2.0.0b (2022-02-08)

  • Initial beta release of V2. V2 is an internal re-write of large parts of the connector to use Databricks edge features. All public APIs from V1 remain.
  • Added Unity Catalog support (pass catalog and / or schema key word args to the .connect method to select initial schema and catalog)

Note: The code for versions prior to v2.0.0b is not contained in this repository. The below entries are included for reference only.

1.0.0 (2022-01-20)

  • Add operations for retrieving metadata
  • Add the ability to access columns by name on result rows
  • Add the ability to provide configuration settings on connect

0.9.4 (2022-01-10)

  • Improved logging and error messages.

0.9.3 (2021-12-08)

  • Add retries for 429 and 503 HTTP responses.

0.9.2 (2021-12-02)

  • (Bug fix) Increased Thrift requirement from 0.10.0 to 0.13.0 as 0.10.0 was in fact incompatible
  • (Bug fix) Fixed error message after query execution failed -SQLSTATE and Error message were misplaced

0.9.1 (2021-09-01)

  • Public Preview release, Experimental tag removed
  • minor updates in internal build/packaging
  • no functional changes

0.9.0 (2021-08-04)