QuotaGuard and Databricks Integration Guide

QuotaGuard and Databricks Integration Guide

QuotaGuard gives your Databricks clients two fixed outbound IP addresses that satisfy a workspace IP access list. Set your client’s proxy settings to point at QuotaGuard, register your two static IPs on an ALLOW list, and every request reaches the workspace from an address the list accepts. This guide covers the REST API, the Python SDK, the CLI, JDBC, ODBC, the SQL connector, Node.js, Go, and dbt, plus testing, the native PrivateLink comparison, and troubleshooting.

This page documents connecting to a Databricks workspace over its public HTTPS interface. It applies wherever IP access lists are enforced.

Why Databricks Connections Need a Static IP

Databricks controls which networks can reach a workspace with IP access lists. When the feature is enabled and an allow list exists, Databricks checks the source IP of every request and blocks anything not on the list. The control covers web application and REST API access to the workspace, which includes the Jobs API, the SQL Statement Execution API, Unity Catalog, the SDK, and the CLI.

Applications on serverless and PaaS hosts use outbound IP addresses that rotate on every deploy, restart, and scale event. There is no fixed address to put on the allow list, and a cloud provider’s published range is far too broad to use for a security allow list.

These clients hit the access list wall most often:

• Backends and APIs on AWS Lambda, Render, Railway, Fly.io, Heroku, and Cloud Functions
• CI and CD pipelines that deploy jobs, run migrations, or call the SQL Statement Execution API
• Orchestration and reverse-ETL tools driving Databricks through the REST API
• BI and SQL clients connecting to SQL warehouses from rotating cloud IPs
• Service integrations and automation built on the Databricks SDK

Two static IPs solve all of these. You register the pair once and the connection stops breaking on every deploy.

Where the IP Access List Is Configured

IP access lists exist at two scopes. Workspace admins configure them on a workspace. Account admins configure them on the account console. Each is an ALLOW or BLOCK list.

You manage them with the IP Access Lists REST API, the Databricks CLI, the account or workspace console, or the SDK. The CLI is the fastest path:

databricks ip-access-lists create quotaguard ALLOW \
  --json '{"ip_addresses": ["<your-static-ip-1>", "<your-static-ip-2>"]}'

Five things to know about how this behaves:

• Lists accept individual IPv4 addresses and CIDR ranges, including a single address as a /32, up to 1000 values across all allow and block lists combined.
• Block lists are evaluated before allow lists. An address on a block list is refused even if it also appears on an allow list.
• The feature requires the Enterprise pricing tier on AWS and Google Cloud, or the Premium plan on Azure, and it supports IPv4 only.
• The feature has to be enabled separately. An allow list has no effect until you turn IP access lists on for the workspace or account.
• Databricks refuses to apply a list that would block the IP of the user setting it, returning a 400 error with error_code INVALID_STATE. This guards that one user, not your compute plane.

Add the public NAT IPs your compute plane uses to reach the control plane to the allow list as well. If you enable the feature without them, you cut off your own clusters.

Databricks is also rolling out context-based ingress control at the account level, which combines identity, request type, and network source into a single policy. Where it is in use, both it and any workspace IP access list apply together, and your QuotaGuard static IPs satisfy the network-source condition the same way.

Getting Started

After creating a QuotaGuard account, you are redirected to your dashboard, where you can find your proxy credentials and two static IP addresses.

Choose the right proxy region: Select the QuotaGuard region closest to your Databricks workspace to minimize latency. The region is set at sign-up. Changes after sign-up require contacting support.

Step 1: Set the Proxy URL

Your QuotaGuard Static connection URL has this shape, with the region host taken from your dashboard:

QUOTAGUARDSTATIC_URL="http://username:password@<your-quotaguard-proxy-host>:9293"

The host is region-specific. Use the exact host shown in your dashboard. The HTTP/HTTPS port is 9293.

Step 2: Route Each Client Through the Proxy

Databricks clients connect over HTTPS, so each one attaches the proxy by pointing at your region host on port 9293. The host, port, and credentials in every example below come from your dashboard and must be exact.

REST API (curl)

Route any REST call through the proxy with the standard proxy flag:

curl -x "http://username:password@<your-quotaguard-proxy-host>:9293" \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  https://dbc-abcd1234-5678.cloud.databricks.com/api/2.1/jobs/list

Python SDK (databricks-sdk)

The SDK honors the standard HTTPS_PROXY environment variable. Set it before your code runs and every call routes through the proxy:

export HTTPS_PROXY="http://username:password@<your-quotaguard-proxy-host>:9293"

from databricks.sdk import WorkspaceClient

w = WorkspaceClient(
    host="https://dbc-abcd1234-5678.cloud.databricks.com",
    token="...",
)

for job in w.jobs.list():
    print(job.settings.name)

Databricks CLI

The CLI is built on the SDK and honors the same environment variable:

export HTTPS_PROXY="http://username:password@<your-quotaguard-proxy-host>:9293"
databricks jobs list

JDBC

The Databricks JDBC driver, version 3 and above, takes the proxy as connection properties in the URL:

jdbc:databricks://dbc-abcd1234-5678.cloud.databricks.com:443/default;httpPath=/sql/1.0/warehouses/abc123;AuthMech=3;UID=token;PWD=<token>;UseProxy=1;ProxyHost=<your-quotaguard-proxy-host>;ProxyPort=9293;ProxyAuth=1;ProxyUID=username;ProxyPWD=password

You can also pass the same properties through a java.util.Properties object instead of the URL.

Cloud Fetch: The JDBC and ODBC drivers use a separate code path called Cloud Fetch to retrieve query results directly from cloud storage. Cloud Fetch has its own proxy properties (UseCFProxy, CFProxyHost, CFProxyPort, CFProxyAuth, CFProxyUID, CFProxyPwd). Routing the main connection through QuotaGuard does not route Cloud Fetch traffic. Most teams leave Cloud Fetch direct so result bandwidth does not consume your QuotaGuard plan; if you must route it through QuotaGuard too, set the CF properties to the same values.

ODBC

The ODBC driver uses the same proxy settings. Set them in the DSN or the connection string:

UseProxy=1;ProxyHost=<your-quotaguard-proxy-host>;ProxyPort=9293;ProxyAuth=1;ProxyUID=username;ProxyPWD=password

Python SQL connector (databricks-sql-connector)

The connector connects over HTTPS. Set HTTPS_PROXY in the environment before you open the connection:

export HTTPS_PROXY="http://username:password@<your-quotaguard-proxy-host>:9293"

from databricks import sql

conn = sql.connect(
    server_hostname="dbc-abcd1234-5678.cloud.databricks.com",
    http_path="/sql/1.0/warehouses/abc123",
    access_token="...",
)

If your connector version does not route through the environment variable, use the JDBC or ODBC driver’s explicit proxy properties instead.

dbt (dbt-databricks)

dbt-databricks runs on the Python SQL connector. Set HTTPS_PROXY in the environment dbt runs in. If your setup does not pick it up, point dbt at the ODBC or JDBC driver and set the proxy properties there.

export HTTPS_PROXY="http://username:password@<your-quotaguard-proxy-host>:9293"

Node.js (@databricks/sql)

The Node client connects over HTTPS. Set HTTPS_PROXY for the process, or configure an HTTPS proxy agent on the client. Confirm the agent option against the current client documentation before relying on it.

Go (databricks-sdk-go)

The Go SDK honors the standard proxy environment variables:

export HTTPS_PROXY="http://username:password@<your-quotaguard-proxy-host>:9293"

Testing the Connection

Confirm two things: that your traffic leaves from a QuotaGuard static IP, and that Databricks sees that IP.

First, check the egress IP through the proxy:

curl -x "http://username:password@<your-quotaguard-proxy-host>:9293" https://ip.quotaguard.com

{"ip":"<one of your two QuotaGuard static IPs>"}

The returned IP must be one of the two static IPs in your dashboard. Call it more than once and you will see both addresses, because the pair is load-balanced.

Second, confirm what Databricks recorded. Databricks audit logs, available in the system.access.audit system table, record the source IP address of each request, so you can verify the address the access list saw:

SELECT event_time, action_name, request_params, source_ip_address
FROM system.access.audit
WHERE source_ip_address IN ('<your-static-ip-1>', '<your-static-ip-2>')
ORDER BY event_time DESC
LIMIT 20;

A row whose source IP matches one of your QuotaGuard IPs confirms the full path.

Databricks PrivateLink vs a NAT Gateway vs QuotaGuard

Databricks’ native answer to the public-IP problem is PrivateLink on AWS, or Private Service Connect on Google Cloud and Private Link on Azure. These remove the public internet path, and IP access lists do not apply to that private traffic. They work and are the right choice for some architectures. They also carry real weight: they require private connectivity on both ends and only apply within the one cloud where your workspace and your client both sit. A self-managed NAT gateway is the other native option, and it only covers the one platform whose VPC you control.

Use PrivateLink or Private Service Connect if your client and workspace are in the same cloud and you want to remove the public path. Use QuotaGuard if you run across clouds or platforms, or want one fixed identity that works everywhere you deploy. This page documents the QuotaGuard path.

Outbound and Inbound

Outbound is the main case for Databricks. Your application, CI pipeline, SDK, or SQL client connects out to the workspace, and Databricks checks the source IP against the access list. Everything above covers this direction.

Inbound applies when traffic flows into your application rather than out to Databricks. If a partner or enterprise system pushes data into your application on a fixed endpoint before your application loads it into Databricks, QuotaGuard includes inbound proxy on Micro plans and above, and QuotaGuard Shield covers the inbound path when that incoming data is regulated.

Troubleshooting

A request is refused with a 403 saying the IP is not allowed The request reached Databricks from an IP that is not on an allow list. Confirm your egress IP with the curl test above, then confirm both QuotaGuard IPs are on an ALLOW list and that the IP access list feature is enabled. If a block list also exists, remember it is evaluated first.

Your own clusters or jobs stop working after enabling the feature The compute plane’s public NAT IPs are not on the allow list. Add them alongside the QuotaGuard pair. The lockout guard only protects the admin who set the list, not your compute plane.

407 Proxy Authentication Required The proxy credentials are wrong or missing. Recheck the username and password against the dashboard. For JDBC and ODBC, confirm ProxyAuth=1 with ProxyUID and ProxyPWD set. A credential with a special character may need URL-encoding.

Connection times out Usually a wrong host or port. Confirm you are using your region’s host from the dashboard and port 9293 for Static.

Wrong IP returned, or only one IP ever appears The pair is load-balanced, so repeated checks show both addresses over time. A non-QuotaGuard IP means the client is not actually using the proxy. Confirm the proxy is set on the client that opens the connection, not on a different process or a global default the runtime ignores.

The allow list change has no effect Two common causes. The feature is not enabled, so the allow list is inert. Or the change has not propagated yet, because list changes take a few minutes to take effect.

An IPv6 address shows up IP access lists are IPv4 only. Make sure your client and proxy path use IPv4 so the address Databricks sees is one it can match.

QuotaGuard Static vs QuotaGuard Shield

Static is right for most Databricks workspaces and satisfies the IP access list on its own. Choose Shield if the workload handles regulated data or the environment requires TLS between your app and the proxy itself.

Ready to Get Started?

Get in touch or create a free trial account.

Try QuotaGuard Now

Contact Support