Metadata-Version: 2.4
Name: structured-data-models-connectors
Version: 1.0.0
Summary: Shared data-source connectors (connect, read, SQL quoting) for the structured-data-models-client and its engines
Author: NVIDIA
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/NVIDIA/structured-data-models-client
Project-URL: Source, https://github.com/NVIDIA/structured-data-models-client
Project-URL: Issues, https://github.com/NVIDIA/structured-data-models-client/issues
Project-URL: Changelog, https://github.com/NVIDIA/structured-data-models-client/blob/main/CHANGELOG.md
Keywords: structured-data,connectors,sql
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: pandas>=2.2.2
Requires-Dist: pyarrow
Provides-Extra: sqlite
Requires-Dist: adbc-driver-sqlite; extra == "sqlite"
Provides-Extra: duckdb
Requires-Dist: duckdb>=1.5.0; extra == "duckdb"
Provides-Extra: snowflake
Requires-Dist: snowflake-connector-python<5,>=4.8; extra == "snowflake"
Provides-Extra: databricks
Requires-Dist: databricks-sql-connector>=4.1.3; extra == "databricks"
Requires-Dist: pyyaml>=6.0.1; extra == "databricks"
Requires-Dist: databricks-sdk<1.0,>=0.89; extra == "databricks"
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.1; extra == "postgres"
Provides-Extra: lakebase
Requires-Dist: psycopg>=3.1; extra == "lakebase"
Requires-Dist: databricks-sdk<1.0,>=0.89; extra == "lakebase"
Provides-Extra: s3
Requires-Dist: s3fs>=2023.12.0; extra == "s3"
Provides-Extra: all
Requires-Dist: structured-data-models-connectors[databricks,duckdb,lakebase,postgres,s3,snowflake,sqlite]; extra == "all"
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: moto[server]>=5; extra == "test"
Dynamic: license-file

# structured-data-models-connectors

Shared data-source connectors for the [`structured-data-models-client`](https://github.com/NVIDIA/structured-data-models-client) client and its
model engines. One place that knows how to reach each warehouse, so the client
(flat table reads) and the Kumo Relational driver (warehouse connections for
its graph samplers) don't each carry their own copy.

It provides:

- `DatabricksSource`, `SnowflakeSource`, `PostgresSource`, `DuckDBSource` and `SQLiteSource`: a connection and the namespace its tables live in, for `Schema.from_source` to sample in place.
  Each takes the driver's own connection, opened with the driver's own `connect`, or a Spark or Snowpark session.
  A source never closes it: the caller owns the connection and closes it.
- `sdm_connectors.databricks.warehouse(warehouse_id, workspace=w)` and `lakebase(endpoint, database=..., workspace=w)`: open a Databricks SQL warehouse or Lakebase connection as the caller's own `WorkspaceClient`.
  Each returns the driver's native connection, for a source to take and the caller to close.
  Databricks requires a partner product to identify every connection and API request it opens, so each names structured-data-models-client as `nvidia_structured-data-models-client/<version>`.
  A warehouse connection carries it in the driver's User-Agent, ahead of any `user_agent_entry` you pass, and tags its statements `structured_data_models_client_purpose: sampling` beside your own query tags, unless you set `structured_data_models_client_purpose` yourself.
  A Lakebase connection carries it as `application_name`, ahead of any you pass, within the 63 bytes Postgres keeps.
  The `WorkspaceClient` requests they make carry it too, through a copy of your client, which is left unchanged.
  A connection or Spark session you open yourself and pass to a source is used as it is, without the identifier or the tag.
- `read(path, *, format=None, storage_options=None) -> DataFrame`: read a CSV or Parquet file as a flat pandas DataFrame, from a local path, a `file:` URI or an `s3://` URI.
- `read_table(connection, table=…, query=…) -> DataFrame`: driver-agnostic fetch over an open connection, in that connection's own session.
- `quote_ident(ident, char='"')` and `resolve_sql(table=…, query=…)`: SQL identifier quoting and a safe table/query guard.

```python
import duckdb
from databricks.sdk import WorkspaceClient
from sdm_connectors import DatabricksSource, DuckDBSource, PostgresSource
from sdm_connectors.databricks import lakebase, warehouse

warehouse_id = '0123456789abcdef'
endpoint = 'projects/<project>/branches/<branch>/endpoints/<endpoint>'

w = WorkspaceClient()
uc = DatabricksSource(
    warehouse(warehouse_id, workspace=w), catalog='main', schema='sales'
)
lb = PostgresSource(
    lakebase(endpoint, database='databricks_postgres', workspace=w),
    schema='public',
)
dd = DuckDBSource(duckdb.connect('sales.duckdb', read_only=True))
```

A file-backed database opened for writing is created empty when its file is missing, and one opened read-only is refused instead.
Open DuckDB with `read_only=True`, and SQLite through a read-only URI such as `adbc_driver_sqlite.dbapi.connect('file:sales.sqlite?mode=ro')`.

File reads accept `.csv` / `.txt` (optionally compressed) and `.parquet` / `.pq` / `.parq`, plus a directory as a Parquet dataset.
Any other suffix is rejected with `INVALID_CONNECTOR_ARGS` rather than parsed as CSV.
Pass `format='csv'` or `format='parquet'` to read a file whose name carries no recognised suffix.
`storage_options` applies to an `s3://` path only, and goes to `s3fs` as it is.

`table=` accepts only plain, unquoted, dot-separated ASCII identifiers, and
interpolates them as written, so they are subject to each backend's default
case folding (Snowflake upper-cases, Databricks lower-cases). A name that needs
quoting (spaces, non-ASCII characters, a leading digit) is rejected with
`INVALID_CONNECTOR_ARGS`. A reserved word such as `select` is a plain
identifier by that rule, so it is accepted here and instead fails at execution
as `QUERY_FAILED`. Either way, reach the table through `query=` with
`quote_ident`, choosing the quote character your backend uses (`"` for
SQLite/DuckDB/Snowflake, `` ` `` for Databricks):

```python
import duckdb
from sdm_connectors import quote_ident, read_table

connection = duckdb.connect('w.db', read_only=True)
read_table(connection, query=f'SELECT * FROM {quote_ident("café")}')
```

Failures raised by the sources, `read`, `read_table` and the Databricks helpers surface as `ConnectorError` with a stable `code`: `UNKNOWN_CONNECTOR`, `INVALID_CONNECTOR_ARGS`, `CONNECT_FAILED`, `QUERY_FAILED`, `READ_FAILED`, or `NOT_FOUND`.
Where a driver raised, its exception is chained as `__cause__`; an unknown connector, and an argument refused before any driver is reached, have none.
A missing driver, or a databricks-sdk too old for `lakebase()`, is refused with code `MISSING_BACKEND`; a missing driver raises it as `MissingBackendError`, which is also a `ModuleNotFoundError`.
A broken or incompatible driver installation stays unwrapped by design and raises `ImportError` (consumers map it to their own broken-install category, e.g. structured-data-models-client's `DRIVER_LOAD_FAILED`).

Pure-python. Database drivers are optional extras:

```bash
pip install "structured-data-models-connectors[sqlite]"      # or [duckdb] / [postgres] / [snowflake] / [databricks] / [lakebase] / [s3] / [all]
```

Consumers depend on it and surface these extras under their own name, e.g.
`structured-data-models-client[snowflake]` and `structured-data-models-relational-engine[databricks]` both resolve the corresponding
`structured-data-models-connectors` extra.

## Local development

```bash
pip install -e ".[sqlite,duckdb,postgres,databricks,test]"
pytest tests
```

The exactness tests check that a value read here survives all the way onto the
wire, so they need `structured-data-models-client` as well. They skip without it; install it too
to run them:

```bash
pip install -e ../structured-data-models-api -e ../structured-data-models-pql -e ../structured-data-models-client
```
