[ZT] WARP external emergency disconnect #26943
+378
−36
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,273 @@ | ||
| --- | ||
| pcx_content_type: how-to | ||
| title: External Emergency Disconnect | ||
| sidebar: | ||
| order: 3 | ||
| badge: Beta | ||
| --- | ||
|
|
||
| import { Details, Render, TabItem, Tabs, APIRequest } from "~/components"; | ||
|
|
||
| <Render file="warp/external-disconnect-availability" product="cloudflare-one" /> | ||
|
|
||
| The External Emergency Disconnect feature allows organizations to remotely disconnect their fleet of WARP clients using their own infrastructure, independent of the standard Cloudflare control plane. This capability mitigates single point of failure risk and ensures continuity of control in case Cloudflare services are unavailable. | ||
|
|
||
| When External Emergency Disconnect is enabled, WARP clients will periodically poll a customer-hosted HTTPS endpoint. A client will only change its connection state if it receives a valid JSON payload with the new state. Any failure to successfully retrieve the state (such as endpoint unreachability, invalid certificate fingerprint, or an improperly structured payload) will not cause a state change on the client. | ||
|
|
||
| You can use External Emergency Disconnect in combination with the dashboard-initiated [Disconnect WARP on all devices](/cloudflare-one/team-and-resources/devices/warp/configure-warp/warp-settings/#disconnect-warp-on-all-devices) setting. A disconnect signal retrieved from the external endpoint will [take precedence](#warp-settings-precedence). | ||
|
|
||
| ## Use cases | ||
|
|
||
| You can use External Emergency Disconnect to address the following scenarios: | ||
|
|
||
| - **Security Incident Response**: Provides the ability to quickly terminate all WARP tunnels across the entire fleet. | ||
| - **Compliance and Auditing**: Fulfills requirements in sensitive or regulated environments that mandate an "emergency stop" capability that is fully isolated, auditable, and controlled by the organization's own on-premises infrastructure. | ||
| - **Disaster Recovery**: If WARP devices cannot reach Cloudflare's API (due to a network outage, routing issue, or client-side misconfiguration), administrators retain the ability to force-disconnect the fleet via the customer-hosted endpoint. | ||
|
|
||
|
|
||
| ## External endpoint requirements | ||
|
|
||
| An external disconnect endpoint is an HTTPS server hosted outside of Cloudflare from which WARP will fetch the emergency disconnect signal. The customer is fully responsible for managing this endpoint. | ||
|
|
||
| ### Endpoint URL | ||
|
|
||
| The external endpoint URL should: | ||
|
|
||
| - Use the HTTPS protocol. | ||
| - Use an IPv4 or IPv6 address as the host, not a domain. | ||
| - Use a public IP to ensure that devices can fetch the latest state regardless of their network location. | ||
|
|
||
| ### Response payload | ||
|
|
||
| The WARP client expects a JSON response payload from the external endpoint with the following format: | ||
|
|
||
| ```json | ||
| { | ||
| "emergency_disconnect": false | true | ||
| } | ||
| ``` | ||
|
|
||
| - If `emergency_disconnect` is set to `true`, the device will initiate an emergency disconnect. | ||
| - If `emergency_disconnect` is set to `false`, the device will continue normal operation. | ||
|
|
||
| ### Cipher suites | ||
|
|
||
| The WARP client establishes a TLS connection using [Rustls](https://github.com/rustls/rustls). Make sure your HTTPS endpoint accepts one of the [cipher suites supported by Rustls](https://docs.rs/rustls/0.21.10/src/rustls/suites.rs.html#125-143). | ||
|
|
||
| ## Set up External Emergency Disconnect | ||
|
|
||
| ### 1. Create an external disconnect endpoint | ||
|
|
||
| To configure External Emergency Disconnect, you will need an HTTPS endpoint in your own infrastructure that serves the global disconnect signal. The WARP client will poll the external endpoint and validate its TLS/SSL certificate against an SHA-256 fingerprint that you upload to Zero Trust. Refer to [External endpoint requirements](#external-endpoint-requirements) for more details. | ||
|
|
||
| The following example demonstrates how to deploy an external disconnect endpoint using an nginx container in Docker. | ||
|
|
||
| 1. Generate a TLS/SSL certificate: | ||
|
|
||
| ```sh | ||
| openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes -keyout key.pem -out cert.pem | ||
| ``` | ||
|
|
||
| You will be prompted to fill in Distinguished Name (DN) fields. Fill in your organization's information or press `Enter` to use the default values. | ||
|
|
||
| The command will output a certificate in PEM format and its private key. Store these files in a secure place. | ||
|
|
||
| 2. Configure an HTTPS server on your network to use this certificate and key: | ||
|
|
||
| a. Create an nginx configuration file called `nginx.conf`: | ||
|
|
||
| ```txt title="nginx.conf" | ||
| events { | ||
| worker_connections 1024; | ||
| } | ||
|
|
||
| http { | ||
| server { | ||
| listen 443 ssl; | ||
| ssl_certificate /certs/cert.pem; | ||
| ssl_certificate_key /certs/key.pem; | ||
| location /status/disconnect { | ||
| default_type application/json; | ||
| return 200 '{"emergency_disconnect": false}'; | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| If needed, replace `/certs/cert.pem` and `/certs/key.pem` with the locations of your certificate and key. | ||
|
|
||
| b. Add the nginx image to your Docker compose file: | ||
|
|
||
| ```yml title="docker-compose.yml" | ||
| services: | ||
| nginx: | ||
| image: nginx:latest | ||
| ports: | ||
| - 3333:443 | ||
| volumes: | ||
| - ./nginx.conf:/etc/nginx/nginx.conf:ro | ||
| - ./certs:/certs:ro | ||
| ``` | ||
|
|
||
| If needed, replace `./nginx.conf` and `./certs` with the locations of your nginx configuration file and certificate. | ||
|
|
||
| c. Start the server: | ||
|
|
||
| ```sh | ||
| docker compose up -d | ||
| ``` | ||
|
|
||
| 3. To test that the HTTPS endpoint is working, run a curl command from the end user's device. You need to pass the `--insecure` option because we are using a self-signed certificate. | ||
|
|
||
| ```sh | ||
| curl --insecure https://<server-ip>:3333/status/disconnect | ||
| ``` | ||
|
|
||
| ```sh output | ||
| {"emergency_disconnect": false} | ||
| ``` | ||
|
|
||
| ### 2. Extract the SHA-256 fingerprint | ||
|
|
||
| <Render file="warp/sha-256-fingerprint" product="cloudflare-one"/> | ||
|
|
||
| ### 3. Turn on External Emergency Disconnect | ||
|
|
||
| <Tabs syncKey="dashPlusAPI"> <TabItem label="Dashboard"> | ||
|
|
||
| To configure External Emergency Disconnect using the dashboard: | ||
|
|
||
| 1. In [Cloudflare One](https://one.dash.cloudflare.com), go to **Team & Resources** > **Devices** > **Management**. | ||
| 2. Select **Global disconnection settings**. | ||
| 3. Find **Manage device connection using an external signal** and select **Edit**. | ||
| 4. Configure the following fields: | ||
| - **Endpoint IP address and port**: Enter the HTTPS URL from which to fetch the external disconnect signal (for example, `https://192.0.2.1:3333/status/disconnect`). The endpoint must use HTTPS and have an IPv4 or IPv6 address as the host. | ||
| - **Polling frequency**: Choose how often WARP should fetch the external disconnect signal. | ||
| - **Certificate fingerprint**: Enter the [SHA-256 fingerprint](#2-extract-the-sha-256-fingerprint) of the HTTPS server certificate (for example, `DD4F4806C57A5BBAF1AA5B080F0541DA75DB468D0A1FE731310149500CCD8662`). | ||
| 5. Select **Save**. | ||
| 6. Turn on **Manage device connection using an external signal**. | ||
|
|
||
| All WARP clients in your organization will now start polling the external endpoint and connect or disconnect based on the response payload. | ||
|
|
||
| </TabItem> | ||
| <TabItem label="API"> | ||
|
|
||
| To configure External Emergency Disconnect using the API, send a `PATCH` request to the `/devices/settings` endpoint: | ||
|
|
||
| <APIRequest | ||
| path="/accounts/{account_id}/devices/settings" | ||
| method="PATCH" | ||
| json={[ | ||
| { | ||
| "external_emergency_signal_enabled": false, | ||
| "external_emergency_signal_url": "https://192.0.2.1:3333/status/disconnect", | ||
| "external_emergency_signal_fingerprint": "DD4F4806C57A5BBAF1AA5B080F0541DA75DB468D0A1FE731310149500CCD8662", | ||
| "external_emergency_signal_interval": "1m" | ||
| } | ||
| ]} | ||
| /> | ||
|
|
||
| </TabItem> | ||
|
|
||
| <TabItem label="MDM"> | ||
|
|
||
| To configure External Emergency Disconnect using an MDM, add the following parameters to your [MDM file](/cloudflare-one/team-and-resources/devices/warp/deployment/mdm-deployment/): | ||
|
|
||
| ```xml | ||
| <key>external_emergency_signal_url</key> | ||
| <string>https://192.0.2.1:3333/status/disconnect</string> | ||
| <key>external_emergency_signal_fingerprint</key> | ||
| <string>DD4F4806C57A5BBAF1AA5B080F0541DA75DB468D0A1FE731310149500CCD8662</string> | ||
| <key>external_emergency_signal_interval</key> | ||
| <integer>60</integer> | ||
| ``` | ||
|
|
||
| </TabItem> | ||
| </Tabs> | ||
|
|
||
| ### 4. Test External Emergency Disconnect | ||
|
|
||
| 1. Ensure that WARP is connected. | ||
| 2. Ensure that the External Emergency Disconnect feature is [turned on](#3-turn-on-external-emergency-disconnect). | ||
| 3. In your [external endpoint](#create-an-external-disconnect-endpoint) configuration, change `emergency_disconnect` to `true`: | ||
| ```json | ||
| {"emergency_disconnect": true} | ||
| ``` | ||
|
|
||
| 4. You may need to reload the server to apply changes. To reload the [example `nginx` server](#create-an-external-disconnect-endpoint): | ||
| ```sh | ||
| docker exec <container-name-or-id> nginx -s reload | ||
| ``` | ||
|
|
||
| WARP will automatically disconnect within the configured polling interval, and the WARP GUI will display [`Admin directed disconnect`](/cloudflare-one/team-and-resources/devices/warp/troubleshooting/client-errors/#admin-directed-disconnect).. To reconnect all devices, change `emergency_disconnect` back to `false`. | ||
|
|
||
| ## View logs | ||
|
|
||
| the global disconnection event will appear in [Admin audit logs] | ||
|
|
||
| (tbd) | ||
|
|
||
| ## Clear External Emergency Disconnect state | ||
|
|
||
| If the external endpoint becomes unavailable or serves an invalid configuration, WARP clients can get stuck in the emergency disconnect state. You can recover clients by removing the External Emergency Disconnect configuration. The method depends on how the feature was originally deployed. | ||
|
|
||
| ### Dashboard or API | ||
|
|
||
| The External Emergency Disconnect state cannot be cleared from the Cloudflare One dashboard. To remove the feature configuration, send a `PATCH` request with the endpoint URL and fingerprint set to empty strings: | ||
|
|
||
| <APIRequest | ||
| path="/accounts/{account_id}/devices/settings" | ||
| method="PATCH" | ||
| json={[ | ||
| { | ||
| "external_emergency_signal_enabled": false, | ||
| "external_emergency_signal_url": "", | ||
| "external_emergency_signal_fingerprint": "", | ||
| "external_emergency_signal_interval": "1m" | ||
| } | ||
| ]} | ||
| /> | ||
|
|
||
| Cloudflare will propagate the new settings to clients, instructing them to stop polling and discard its cached emergency state. | ||
|
|
||
| ### MDM | ||
|
|
||
| If you deployed External Emergency Disconnect using an MDM, remove the `external_emergency_signal_url` key (and other related keys) from the MDM profile. Then, use your MDM to push the changes to devices. WARP will stop polling the external endpoint and discard its cached emergency state. | ||
|
|
||
| Alternatively, users can switch WARP to a [different MDM configuration](/cloudflare-one/team-and-resources/devices/warp/deployment/mdm-deployment/switch-organizations/) that does not have the feature configured. | ||
|
|
||
| ### Local client reset | ||
|
|
||
| As a last resort, you can use the CLI to reset External Emergency Disconnect on an individual device: | ||
|
|
||
| ``` | ||
| warp-cli registration delete | ||
| ``` | ||
|
|
||
| This command will clear the client registration, clear the local policy, and discard the cached emergency state. Users will need to [re-enroll WARP](/cloudflare-one/team-and-resources/devices/warp/deployment/manual-deployment/) with your Zero Trust organization. | ||
|
|
||
| ## WARP settings precedence | ||
|
|
||
| Learn how global disconnect settings interact and how they impact other WARP profile settings. | ||
|
|
||
| ### Global disconnection settings | ||
|
|
||
| The client will honor disconnect signals from both the Cloudflare dashboard (via [Disconnect WARP on all devices](/cloudflare-one/team-and-resources/devices/warp/configure-warp/warp-settings/#disconnect-warp-on-all-devices)) and the external endpoint. A global disconnect is enforced if either source triggers it. | ||
|
|
||
| | | [Disconnect WARP on all devices](/cloudflare-one/team-and-resources/devices/warp/configure-warp/warp-settings/#disconnect-warp-on-all-devices) is `On` | [Disconnect WARP on all devices](/cloudflare-one/team-and-resources/devices/warp/configure-warp/warp-settings/#disconnect-warp-on-all-devices) is `Off` | | ||
| | --- | --- | --- | | ||
| | **External endpoint returns `true`** | Force disconnected | Force disconnected | | ||
| | **External endpoint returns `false`** | Force disconnected | Normal operation | | ||
|
|
||
| ### Auto connect | ||
|
|
||
| [Auto connect](/cloudflare-one/team-and-resources/devices/warp/configure-warp/warp-settings/#auto-connect) does not apply while a global disconnect is in effect. | ||
|
|
||
| ### Lock WARP switch | ||
|
|
||
| [Lock WARP switch](/cloudflare-one/team-and-resources/devices/warp/configure-warp/warp-settings/#lock-warp-switch) does not apply while a global disconnect is in effect. Users will be unable to turn on WARP unless they have an [admin override code](#admin-override). | ||
|
|
||
| ### Admin override | ||
|
|
||
| A global disconnect will clear any existing [admin override codes](#allow-admin-override-codes). The only way for users to reconnect during a global disconnect is by using a new [admin override code](#allow-admin-override-codes). For example, you may want to provide IT staff with a code so that they can test resolution of the incident that led to the global disconnect. The override code will exempt a specific user and device from the global disconnect until the override timeout expires. | ||
|
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
need info on how an external disconnect shows up in dashboard Audit logs.