From 03477b8877252b72e9fea6e7a9d31adffeae1711 Mon Sep 17 00:00:00 2001 From: Enrique Gonzalez Date: Thu, 8 Oct 2026 01:51:34 +0000 Subject: [PATCH 1/4] docs: Update observability troubleshooting guide for 8.0 Amp-Thread-ID: https://ampcode.com/threads/T-726be5c1-be6e-4eaf-9008-a0e55c66fed4 Co-authored-by: Amp --- .../observability/troubleshooting.mdx | 133 +++++++++--------- 1 file changed, 68 insertions(+), 65 deletions(-) diff --git a/docs/self-hosted/observability/troubleshooting.mdx b/docs/self-hosted/observability/troubleshooting.mdx index 29a05366a..8a05dc4f7 100644 --- a/docs/self-hosted/observability/troubleshooting.mdx +++ b/docs/self-hosted/observability/troubleshooting.mdx @@ -22,26 +22,27 @@ issue or generating a high-quality issue report. If this is the case, this could indicate high gitserver load. To confirm, take the following steps: -1. [Open Grafana](/self-hosted/observability/metrics#grafana). -1. **If using Sourcegraph 3.14**: Simply check if either of these alerts are firing: - - `gitserver: 50+ concurrent command executions (abnormally high load)` - - `gitserver: echo command execution duration exceeding 1s` -1. **If using an older version of Sourcegraph:** - - Go to the Sourcegraph Internal > Gitserver rev2 dashboard. - - Examine the "Echo Duration Seconds" dashboard (tracks the `src_gitserver_echo_duration_seconds` - metric) and "Commands running concurrently" dashboard (tracks the `src_gitserver_exec_running` - metric). If either of these is high (> 1s echo duration or 100s simultaneous execs), then this - indicates gitserver is under heavy load and likely the bottleneck. +1. [Open Grafana](/self-hosted/observability/metrics#grafana) and go to the **Git Server** dashboard. +1. Check whether the [`gitserver: echo_command_duration_test`](/self-hosted/observability/alerts#gitserver-echo_command_duration_test) + alert is firing. The **Echo test command duration** panel tracks the + `src_gitserver_echo_duration_seconds` metric; on a healthy node it stays below 5ms, and the alert + fires when it stays at 20ms or more for 30 seconds. +1. Check the **Git commands running on each gitserver instance** panel, which tracks the + `src_gitserver_exec_running` metric. A consistently high value indicates gitserver is under + heavy load and likely the bottleneck. 1. Confirm your gitserver is not under-provisioned, by e.g. comparing its allocated resources with what the [resource estimator](/self-hosted/deploy/resource-estimator) shows. -Solution: set `USE_ENHANCED_LANGUAGE_DETECTION=false` in the Sourcegraph runtime -environment. +If gitserver is overloaded, follow the next steps for the +[`echo_command_duration_test`](/self-hosted/observability/alerts#gitserver-echo_command_duration_test) +alert: allocate gitserver CPU and memory according to the resource estimator, and provision more +IOPS if its persistent volume is slow. #### Scenario: no cloning, syncing, updating or deleting is happening Observed state: Sourcegraph instance does not react to any updates to code hosts and no cloning is happening. The cause of this state could be worker queries that are too large for the limits of the running Postgres DB. -One symptom is seeing repo syncer errors ending in `driver: bad connection` in the worker logs, or the same error in the "Code host status panel" (Clicking the cloud icon). +One symptom is seeing repo syncer errors ending in `driver: bad connection` in the worker logs, or the same error in the +status notifications in the navigation bar and on the **Site admin > Code host connections** page (`/admin/external-services`). The fix is to increase the memory on Postgres DB which will increase certain Postgres-internal limits and will allow the queries from worker to go through. @@ -99,9 +100,10 @@ find a repro if possible. If that isn't possible, file an issue with the followi 1. Open the [browser developer network panel](#check-browser-network-panel) and identify slow requests. 1. [Use Jaeger](#collect-a-jaeger-trace) to drill down into slow requests and understand which - components of the request are slow. Remember that many Sourcegraph API requests identify the - Jaeger trace ID in the `x-trace` HTTP response header, which makes it easy to look up the trace - corresponding to a particular request. + components of the request are slow. Sourcegraph returns the trace ID in the `x-trace` HTTP + response header and, when a tracing backend is configured, a link to the trace in the + `x-trace-url` header, which makes it easy to look up the trace corresponding to a particular + request. 1. Copy the [Sourcegraph configuration](#copy-configuration) to the error report. #### Scenario: the issue is performance-related and there is NOT a consistent reproduction @@ -158,21 +160,18 @@ If Sourcegraph feels sluggish overall, the likely culprit is resource allocation If you are seeing cAdvisor metrics from a namespace outside of the one Sourcegraph is currently deployed into. -1. Update your cAdvisor configuration to use namespaced monitoring. See the [cAdvisor monitoring example](https://github.com/sourcegraph/deploy-sourcegraph-k8s/tree/main/examples/monitoring/cadvisor) and [cAdvisor README](https://github.com/sourcegraph/deploy-sourcegraph-k8s/blob/main/base/monitoring/cadvisor/README.md) for namespace configuration +1. Update your Prometheus configuration to keep only metrics from the Sourcegraph namespace. See + [Filtering cAdvisor metrics](/self-hosted/deploy/kubernetes/configure#filtering-cadvisor-metrics). 2. Apply this configuration and restart Prometheus -_Note:_ This is unneeded if you are using the 'namespaced' overlay - #### Scenario: search timeouts If your users are experiencing search timeouts or search performance issues, please perform the following steps: -1. Try appending variations of `index:only`, `timeout:60s` and `count:999999` to the search query to see if it is still slow. -1. [Access Grafana directly](/self-hosted/observability/metrics#accessing-grafana-directly). -1. Select the **+** icon on the left-hand side, then choose **Import**. -1. Paste [this JSON](https://gist.githubusercontent.com/slimsag/3fcc134f5ce09728188b94b463131527/raw/f8b545f4ce14b0c30a93f05cd1ee469594957a2c/sourcegraph-debug-search-timeouts.json) into the input and click **Load**. -1. Once the dashboard appears, include screenshots of **the entire** dashboard in the issue report. -1. Include the logs of `zoekt-webserver` container in the `indexed-search` pods. If you are using single Docker container enable [debug logs](/self-hosted/observability/#Logs) first. +1. Try appending variations of `index:only`, `timeout:60s` and `count:all` to the search query to see if it is still slow. +1. [Trace the search query](/self-hosted/observability/tracing#trace-a-search-query) and include the trace in the issue report. +1. [Open Grafana](/self-hosted/observability/metrics#grafana) and include screenshots of the **Frontend** dashboard's **Search at a glance** section and of the **Zoekt** dashboard in the issue report. +1. Include the [logs](#examine-logs) of the `zoekt-webserver` container in the `indexed-search` pods (Kubernetes) or the `zoekt-webserver-0` container (Docker Compose). #### Scenario: zoekt-webserver is in a `CrashloopBackOff` and `err cannot allocate memory` @@ -198,7 +197,7 @@ $ sysctl -n vm.max_map_count - 250,000 repositories / 2 indexed-search replicas = 125,000 repos to index per replica. -1. If the `vm.max_map_count` is lower than the result of the above calculation. Adjust the `vm.max_map_count` by executing: +1. If the `vm.max_map_count` is lower than twice the result of the above calculation (250,000 in this example), raise the `vm.max_map_count` to at least that value, for example: sudo sysctl -w vm.max_map_count=262144 @@ -211,6 +210,11 @@ $ sudo sysctl -n vm.max_map_count 1. Ensure the change will persist a system reboot by updating the `vm.max_map_count` setting in `/etc/sysctl.conf`. +The [`zoekt: memory_map_areas_percentage_used`](/self-hosted/observability/alerts#zoekt-memory_map_areas_percentage_used) +alert warns before Zoekt runs out of memory map areas. Adding `indexed-search` replicas or keeping +[shard merging](/admin/search#shard-merging) enabled (the default) also reduces the number of memory +map areas each replica needs. + #### Scenario: zoekt-webserver is restarting due to watchdog zoekt-webserver has a built in watchdog which ensures it can respond to search requests. If the watchdog fails, it will panic causing the process to exit with a non-zero exit code. This is like a Kubernetes health check, but works across all our deployment environments. @@ -256,38 +260,34 @@ Network panel](https://developers.google.com/web/tools/chrome-devtools/network). - Check the waterfall diagram at the top and the Waterfall column in the list of network requests to quickly identify high-latency requests. - Clicking on a request will open up a panel that provides additional details about the request. - - If a GraphQL request is taking a long time, you should obtain its Jaeger trace ID by inspecting - the Headers tab of this panel and finding the `X-Trace` or `x-trace` response header value. Once - you've obtained this trace ID, [look it up in Jaeger](#collect-a-jaeger-trace). + - If a request is taking a long time, inspect the Headers tab of this panel and find the + `x-trace` response header (the trace ID) or the `x-trace-url` response header (a link to the + trace). Once you've obtained the trace ID, [look it up in Jaeger](#collect-a-jaeger-trace). - You can check `Preserve log` to preserve the list of requests across page loads and reloads. ### Check resource usage -[Access Prometheus](/self-hosted/observability/metrics#accessing-prometheus) and examine the following metrics: - -**Memory:** `process_resident_memory_bytes` is a gauge that tracks memory usage per backend process. +[Open Grafana](/self-hosted/observability/metrics#grafana). Most service dashboards (for example +**Frontend**, **Git Server**, and **Searcher**) have these sections: -- Example: `process_resident_memory_bytes{app="indexed-search"}` shows memory usage for each - indexed-search instance. +- **Container monitoring**: CPU and memory usage, filesystem I/O, and containers that went missing + (for example, because they were OOM killed). +- **Provisioning indicators**: short-term and long-term CPU and memory usage compared with the + container's limits, which shows whether a service is under-provisioned. -**CPU:** `process_cpu_seconds_total` is a counter that tracks cumulative CPU seconds used . +**Disk:** the **Git Server** dashboard's **Disk space remaining** panel tracks free disk space on +gitserver, from the `src_gitserver_disk_space_available` and `src_gitserver_disk_space_total` metrics. -- Example: `rate(process_cpu_seconds_total{app="sourcegraph-frontend"}[1m])` shows average CPU usage - for each sourcegraph-frontend instance over the last minute. - -**Disk:** `gitserver_disk_free_percent` is a gauge that tracks free disk space on gitserver. +To query the underlying metrics yourself, [access Prometheus directly](/self-hosted/observability/metrics#accessing-prometheus-directly). ### Check end-user stats -Go to `/site-admin/usage-statistics` to view daily, weekly, and monthly user statistics. - -To drill down (e.g., into sub-daily traffic, visits per page type, latencies, etc.), [access -Grafana](/self-hosted/observability/metrics#grafana) and visit the Sourcegraph Internal > HTTP dashboard page, which -includes the following panels: +Use [Sourcegraph Analytics](/analytics) to view usage over time, such as active users and the +number of searches. -- QPS by Status Code -- QPS by URL Route -- P90 Request Duration (request latency at the 90th percentile) +To drill down into search traffic and latencies, [open Grafana](/self-hosted/observability/metrics#grafana) +and go to the **Frontend** dashboard. Its **Search at a glance** section includes the 90th and 99th +percentile search request duration, search timeouts, and search errors. Grafana contains ready-made dashboards derived from Prometheus metrics. Any chart in Grafana mentioned here can be viewed in Prometheus by clicking the dropdown menu next to the Grafana panel @@ -295,33 +295,34 @@ title > Edit > copying the expression in the Metrics field. ### Check error rates -[Access Grafana](/self-hosted/observability/metrics#grafana) and view the following charts: +[Open Grafana](/self-hosted/observability/metrics#grafana) and view the following dashboards: -- Folder: Sourcegraph Internal > Dashboard: HTTP > Chart: QPS by Status Code - - This shows request rates by HTTP status code for end-user requests. -- Folder: General - - This contains dashboards for each core service in Sourcegraph. Examine each for high-level - metrics important to the health of each service. +- **Overview**: a summary of alert events across all Sourcegraph services. +- **Frontend**: the **Search at a glance** section shows search timeouts and errors for end-user + requests. +- Each service dashboard (for example **Git Server**, **Searcher**, and **Zoekt**) shows + high-level metrics important to the health of that service, along with its alert events. ### Collect a Jaeger trace If you are looking for the trace associated with a specific request, - [Find the trace ID in the HTTP response in the browser developer tools "Network" tab](#check-browser-network-panel). -- [Access Jaeger](/self-hosted/observability/tracing#accessing-jaeger) and look up the trace ID. +- [Open Jaeger](/self-hosted/observability/tracing#jaeger) and look up the trace ID. If you do not have a specific request or cannot find the trace ID, -- [Access Jaeger](/self-hosted/observability/tracing#accessing-jaeger). +- [Open Jaeger](/self-hosted/observability/tracing#jaeger). - Search for a matching span by setting the appropriate fields in the sidebar. -2 ways: start with a span ID, or manually locate your span by searching the Jaeger GUI +See [How to use traces](/self-hosted/observability/tracing#how-to-use-traces) for how to request a +trace for a search query or GraphQL request. ### Examine logs -If you are using the single Docker container or Docker Compose deployment option, logs are printed -to `stdout` and `stderr`. You should be able to access these using your infrastructure provider's -standard log viewing mechanism. +If you are using the Docker Compose deployment option, logs are printed to `stdout` and `stderr`. +Retrieve them with `docker logs $CONTAINER_NAME` (for example, `docker logs sourcegraph-frontend-0`), +or use your infrastructure provider's standard log viewing mechanism. If you are using Kubernetes, @@ -332,17 +333,19 @@ $POD_ID`. This can be useful for diagnosing why a container crashed. - You can tail logs for all pods associated with a given deployment: `kubectl logs -f deployment/sourcegraph-frontend --container=frontend --since=10m` +See [Logs](/self-hosted/observability/logs) to change the log level or format. + ### Copy configuration -Go to the URL path `/site-admin/report-bug` to obtain an all-in-one text box of all Sourcegraph -configuration (which includes advanced configuration, code host configuration, and global -settings). This lets you easily copy all configuration to an issue report (NOTE: remember to redact -any secrets). +Go to **Settings > Report a bug** (the URL path `/user/settings/report-bug`) to obtain an all-in-one +text box of all Sourcegraph configuration. For site admins, this includes the site configuration, +code host configuration, and settings. This lets you easily copy all configuration to an issue +report (NOTE: remember to redact any secrets). ### Collect instance stats The following statistics are useful background context when reporting a performance issue: -- Number of repositories (can be found on the `/site-admin/repositories` page, search for "repositories total") +- Number of repositories (shown at the top of the **Site admin > Repositories** page, `/admin/repositories`) - Size distribution of repositories (e.g., are there one or more large "monorepos" that contain most of the code?) -- Number of users and daily usage stats from `/site-admin/usage-statistics` +- Number of users and usage stats from [Sourcegraph Analytics](/analytics) From a3772816f2d942d36c0a406f5e95390e2a1730e6 Mon Sep 17 00:00:00 2001 From: Enrique Gonzalez Date: Thu, 8 Oct 2026 03:01:39 +0000 Subject: [PATCH 2/4] docs: Add Sourcegraph Cloud troubleshooting checklist and Diagnostics action Amp-Thread-ID: https://ampcode.com/threads/T-726be5c1-be6e-4eaf-9008-a0e55c66fed4 Co-authored-by: Amp --- docs/cloud/index.mdx | 3 +- .../observability/troubleshooting.mdx | 59 +++++++++++++++++-- 2 files changed, 55 insertions(+), 7 deletions(-) diff --git a/docs/cloud/index.mdx b/docs/cloud/index.mdx index 13b6fd35f..7a4806e1f 100644 --- a/docs/cloud/index.mdx +++ b/docs/cloud/index.mdx @@ -120,7 +120,8 @@ For unsupported private connectivity methods, Sourcegraph offers connectivity vi ### Health monitoring, support, and SLAs -- Instance performance and health [monitored](/self-hosted/observability/) by our team's on-call engineers. +- Our on-call engineers monitor instance performance and health. +- For errors or performance issues, follow the [Sourcegraph Cloud troubleshooting checklist](/self-hosted/observability/troubleshooting#sourcegraph-cloud). - [Support and SLAs](/sla/#sourcegraph-cloud-sla-managed-instance). ### Backup and restore diff --git a/docs/self-hosted/observability/troubleshooting.mdx b/docs/self-hosted/observability/troubleshooting.mdx index 8a05dc4f7..3ced6152d 100644 --- a/docs/self-hosted/observability/troubleshooting.mdx +++ b/docs/self-hosted/observability/troubleshooting.mdx @@ -3,7 +3,13 @@ This is a guide for investigating errors and performance issues, with the goal of resolving the issue or generating a high-quality issue report. -**How to use this guide:** + + **Sourcegraph Cloud:** follow the [Sourcegraph Cloud checklist](#sourcegraph-cloud). The scenarios + and the Grafana, Prometheus, Jaeger, and log actions below are for self-hosted deployments: Cloud + customers cannot access them, and Sourcegraph operators investigate them on your behalf. + + +**How to use this guide for self-hosted deployments:** 1. Scan through [Specific scenarios](#specific-scenarios) to see if any of these applies to you. If any do, follow the instructions in the subsection. @@ -16,6 +22,35 @@ issue or generating a high-quality issue report. > NOTE: This guide assumes you have administrator privileges on the Sourcegraph instance. If you are > non-admin user, report your error the administrator and have them walk through this guide. +## Sourcegraph Cloud + +Sourcegraph manages the infrastructure of your Cloud instance. To report an error or performance +issue, collect what you can from the following list and +[contact Sourcegraph Support](https://help.sourcegraph.com/hc/en-us/requests/new) or email +`support@sourcegraph.com`. Don't delay contacting support if some of this information is +unavailable. + +1. **Context:** your instance URL, the affected users and repositories, when the issue occurred + (with time zone), how often it happens, reproduction steps, expected and actual behavior, and + screenshots or the exact error messages. +1. **Browser evidence:** the [browser console](#check-browser-console) output, and the slow or + failing requests from the [browser network panel](#check-browser-network-panel), with their + `x-trace` response header when present. Send the trace ID to support; the Jaeger link in the + `x-trace-url` header is not accessible to Cloud customers. +1. **Search issues:** the original query, and whether adding `index:only`, `timeout:60s`, or + `count:all` to it changes the result. To get a trace ID for a search, add the `trace=1` + parameter to the search results page URL (`&trace=1` if the URL already has parameters), then + copy the `x-trace` header as above. +1. **Repository syncing issues:** errors shown in the status notifications in the navigation bar and + on the **Site admin > Code host connections** page (`/admin/external-services`), and the + affected repositories. +1. **Site admin evidence:** relevant findings from [Diagnostics](#check-diagnostics), your + [Sourcegraph configuration](#copy-configuration), and [instance stats](#collect-instance-stats). + Redact credentials, tokens, and other sensitive content before sharing. + +Audit logs are separate from the server logs Sourcegraph uses for troubleshooting, and are +[available on request or through LogPush](/cloud#audit-logs). + ## Specific scenarios #### Scenario: search _and_ code pages take a long time to load @@ -54,7 +89,8 @@ to be updated it could be from `Out of memory` errors. A fix here is to increase This section contains a list of scenarios, each of which contains instructions that include [actions](#actions) that are appropriate to the scenario. Note that more than one scenario may apply -to a given issue. +to a given issue. For any scenario, [Diagnostics](#check-diagnostics) may show related failures or +slow requests. #### Scenario: the issue is NOT performance-related and there is a consistent reproduction. @@ -242,7 +278,8 @@ You can further diagnose an overloaded zoekt-webserver via watchdog logs or metr This section contains various actions that can be taken to collect information or update Sourcegraph in order to resolve an error or performance issue. You should typically not read this section directly, but start with the [General scenarios](#general-scenarios) section to determine which actions are -appropriate. +appropriate. On Sourcegraph Cloud, start with the [Sourcegraph Cloud checklist](#sourcegraph-cloud) +instead. ### Check browser console @@ -260,11 +297,21 @@ Network panel](https://developers.google.com/web/tools/chrome-devtools/network). - Check the waterfall diagram at the top and the Waterfall column in the list of network requests to quickly identify high-latency requests. - Clicking on a request will open up a panel that provides additional details about the request. - - If a request is taking a long time, inspect the Headers tab of this panel and find the - `x-trace` response header (the trace ID) or the `x-trace-url` response header (a link to the - trace). Once you've obtained the trace ID, [look it up in Jaeger](#collect-a-jaeger-trace). + - If a request is taking a long time, inspect the Headers tab of this panel and copy the + `x-trace` response header (the trace ID) when present. On self-hosted deployments, the + `x-trace-url` response header links to the trace, or you can + [look up the trace ID in Jaeger](#collect-a-jaeger-trace). On Sourcegraph Cloud, send the + trace ID and the time of the request to support; Cloud customers cannot open the Jaeger link. - You can check `Preserve log` to preserve the list of requests across page loads and reloads. +### Check diagnostics + +Site admins on self-hosted and Sourcegraph Cloud instances can go to **Site admin > Diagnostics** +(`/admin/diagnostics`). Review its **Background jobs**, **Outbound requests**, **Pings**, and +**Slow requests** tabs for failures or slow activity related to the issue, and include the relevant +error details or screenshots in the issue report. The **Migrations** tab is not available to Cloud +customers. + ### Check resource usage [Open Grafana](/self-hosted/observability/metrics#grafana). Most service dashboards (for example From 8b2fd3b4a76414aa958de45a20a755e747b8e306 Mon Sep 17 00:00:00 2001 From: Enrique Gonzalez Date: Thu, 8 Oct 2026 17:11:00 +0000 Subject: [PATCH 3/4] docs: Move Sourcegraph Cloud troubleshooting to its own page Amp-Thread-ID: https://ampcode.com/threads/T-726be5c1-be6e-4eaf-9008-a0e55c66fed4 Co-authored-by: Amp --- docs/cloud/index.mdx | 2 +- docs/cloud/troubleshooting.mdx | 71 +++++++++++++++++++ .../observability/troubleshooting.mdx | 56 +++------------ src/data/navigation.ts | 8 ++- 4 files changed, 90 insertions(+), 47 deletions(-) create mode 100644 docs/cloud/troubleshooting.mdx diff --git a/docs/cloud/index.mdx b/docs/cloud/index.mdx index 7a4806e1f..60859fd0a 100644 --- a/docs/cloud/index.mdx +++ b/docs/cloud/index.mdx @@ -121,7 +121,7 @@ For unsupported private connectivity methods, Sourcegraph offers connectivity vi ### Health monitoring, support, and SLAs - Our on-call engineers monitor instance performance and health. -- For errors or performance issues, follow the [Sourcegraph Cloud troubleshooting checklist](/self-hosted/observability/troubleshooting#sourcegraph-cloud). +- For errors or performance issues, follow the [Sourcegraph Cloud troubleshooting guide](/cloud/troubleshooting). - [Support and SLAs](/sla/#sourcegraph-cloud-sla-managed-instance). ### Backup and restore diff --git a/docs/cloud/troubleshooting.mdx b/docs/cloud/troubleshooting.mdx new file mode 100644 index 000000000..539683dac --- /dev/null +++ b/docs/cloud/troubleshooting.mdx @@ -0,0 +1,71 @@ +# Troubleshooting Sourcegraph Cloud + +Sourcegraph manages the infrastructure of your Cloud instance, and our on-call engineers monitor its +performance and health. Site admins on Cloud instances do not have access to Grafana, Prometheus, +Jaeger, or server logs; Sourcegraph investigates those on your behalf. + +To report an error or performance issue, collect what you can from the sections below and +[contact Sourcegraph Support](https://help.sourcegraph.com/hc/en-us/requests/new) or email +`support@sourcegraph.com`. Don't delay contacting support if some of this information is +unavailable. See [Support and SLAs](/sla#sourcegraph-cloud-sla-managed-instance) for response times. + +## Describe the issue + +Include: + +- Your instance URL +- The affected users and repositories +- When the issue occurred (with time zone) and how often it happens +- Reproduction steps, and the expected and actual behavior +- Screenshots or the exact error messages + +## Collect browser evidence + +Open the browser developer tools (right-click in the browser > **Inspect**): + +- **Console** tab: copy any errors shown while reproducing the issue. +- **Network** tab: find slow or failing requests. Select a request and, in its **Headers** tab, copy + the `x-trace` response header when present. Send this trace ID and the time of the request to + support. The Jaeger link in the `x-trace-url` header is only accessible to Sourcegraph. + +Check **Preserve log** to keep the list of requests across page loads. + +## Search issues + +Include the original query, and whether adding `index:only`, `timeout:60s`, or `count:all` to it +changes the result. + +To get a trace ID for a search, add the `trace=1` parameter to the search results page URL +(`&trace=1` if the URL already has parameters), run the search, and copy the `x-trace` header from +the [Network tab](#collect-browser-evidence). + +## Repository syncing issues + +If repositories are not cloning or updating, include the errors shown in the status notifications in +the navigation bar and on the **Site admin > Code host connections** page +(`/admin/external-services`), and the affected repositories. + +## Check diagnostics + +Go to **Site admin > Diagnostics** (`/admin/diagnostics`). Review its **Background jobs**, +**Outbound requests**, **Pings**, and **Slow requests** tabs for failures or slow activity related +to the issue, and include the relevant error details or screenshots. + +## Copy configuration + +Go to **Settings > Report a bug** (`/user/settings/report-bug`) and copy the configuration shown +there. For site admins, it includes the site configuration, code host configuration, and settings. +Redact credentials, tokens, and other sensitive content before sharing. + +## Collect instance stats + +The following are useful context for performance issues: + +- Number of repositories (shown at the top of the **Site admin > Repositories** page, `/admin/repositories`) +- Whether one or more large monorepos contain most of the code +- Number of users and usage stats from [Sourcegraph Analytics](/analytics) + +## Audit logs + +Audit logs are separate from the server logs Sourcegraph uses for troubleshooting. They are +[available on request or through LogPush](/cloud#audit-logs). diff --git a/docs/self-hosted/observability/troubleshooting.mdx b/docs/self-hosted/observability/troubleshooting.mdx index 3ced6152d..5ab27090d 100644 --- a/docs/self-hosted/observability/troubleshooting.mdx +++ b/docs/self-hosted/observability/troubleshooting.mdx @@ -4,12 +4,11 @@ This is a guide for investigating errors and performance issues, with the goal o issue or generating a high-quality issue report. - **Sourcegraph Cloud:** follow the [Sourcegraph Cloud checklist](#sourcegraph-cloud). The scenarios - and the Grafana, Prometheus, Jaeger, and log actions below are for self-hosted deployments: Cloud - customers cannot access them, and Sourcegraph operators investigate them on your behalf. + This guide is for self-hosted deployments. If you use Sourcegraph Cloud, see + [Troubleshooting Sourcegraph Cloud](/cloud/troubleshooting). -**How to use this guide for self-hosted deployments:** +**How to use this guide:** 1. Scan through [Specific scenarios](#specific-scenarios) to see if any of these applies to you. If any do, follow the instructions in the subsection. @@ -22,35 +21,6 @@ issue or generating a high-quality issue report. > NOTE: This guide assumes you have administrator privileges on the Sourcegraph instance. If you are > non-admin user, report your error the administrator and have them walk through this guide. -## Sourcegraph Cloud - -Sourcegraph manages the infrastructure of your Cloud instance. To report an error or performance -issue, collect what you can from the following list and -[contact Sourcegraph Support](https://help.sourcegraph.com/hc/en-us/requests/new) or email -`support@sourcegraph.com`. Don't delay contacting support if some of this information is -unavailable. - -1. **Context:** your instance URL, the affected users and repositories, when the issue occurred - (with time zone), how often it happens, reproduction steps, expected and actual behavior, and - screenshots or the exact error messages. -1. **Browser evidence:** the [browser console](#check-browser-console) output, and the slow or - failing requests from the [browser network panel](#check-browser-network-panel), with their - `x-trace` response header when present. Send the trace ID to support; the Jaeger link in the - `x-trace-url` header is not accessible to Cloud customers. -1. **Search issues:** the original query, and whether adding `index:only`, `timeout:60s`, or - `count:all` to it changes the result. To get a trace ID for a search, add the `trace=1` - parameter to the search results page URL (`&trace=1` if the URL already has parameters), then - copy the `x-trace` header as above. -1. **Repository syncing issues:** errors shown in the status notifications in the navigation bar and - on the **Site admin > Code host connections** page (`/admin/external-services`), and the - affected repositories. -1. **Site admin evidence:** relevant findings from [Diagnostics](#check-diagnostics), your - [Sourcegraph configuration](#copy-configuration), and [instance stats](#collect-instance-stats). - Redact credentials, tokens, and other sensitive content before sharing. - -Audit logs are separate from the server logs Sourcegraph uses for troubleshooting, and are -[available on request or through LogPush](/cloud#audit-logs). - ## Specific scenarios #### Scenario: search _and_ code pages take a long time to load @@ -278,8 +248,7 @@ You can further diagnose an overloaded zoekt-webserver via watchdog logs or metr This section contains various actions that can be taken to collect information or update Sourcegraph in order to resolve an error or performance issue. You should typically not read this section directly, but start with the [General scenarios](#general-scenarios) section to determine which actions are -appropriate. On Sourcegraph Cloud, start with the [Sourcegraph Cloud checklist](#sourcegraph-cloud) -instead. +appropriate. ### Check browser console @@ -297,20 +266,17 @@ Network panel](https://developers.google.com/web/tools/chrome-devtools/network). - Check the waterfall diagram at the top and the Waterfall column in the list of network requests to quickly identify high-latency requests. - Clicking on a request will open up a panel that provides additional details about the request. - - If a request is taking a long time, inspect the Headers tab of this panel and copy the - `x-trace` response header (the trace ID) when present. On self-hosted deployments, the - `x-trace-url` response header links to the trace, or you can - [look up the trace ID in Jaeger](#collect-a-jaeger-trace). On Sourcegraph Cloud, send the - trace ID and the time of the request to support; Cloud customers cannot open the Jaeger link. + - If a request is taking a long time, inspect the Headers tab of this panel and find the + `x-trace` response header (the trace ID) or the `x-trace-url` response header (a link to the + trace). Once you've obtained the trace ID, [look it up in Jaeger](#collect-a-jaeger-trace). - You can check `Preserve log` to preserve the list of requests across page loads and reloads. ### Check diagnostics -Site admins on self-hosted and Sourcegraph Cloud instances can go to **Site admin > Diagnostics** -(`/admin/diagnostics`). Review its **Background jobs**, **Outbound requests**, **Pings**, and -**Slow requests** tabs for failures or slow activity related to the issue, and include the relevant -error details or screenshots in the issue report. The **Migrations** tab is not available to Cloud -customers. +Go to **Site admin > Diagnostics** (`/admin/diagnostics`). Review its **Background jobs**, +**Outbound requests**, **Pings**, **Slow requests**, and **Migrations** tabs for failures or slow +activity related to the issue, and include the relevant error details or screenshots in the issue +report. ### Check resource usage diff --git a/src/data/navigation.ts b/src/data/navigation.ts index 62fb72b45..c56c1d971 100644 --- a/src/data/navigation.ts +++ b/src/data/navigation.ts @@ -568,7 +568,13 @@ export const navigation: NavigationItem[] = [ }, { title: 'Sourcegraph Cloud', - href: '/cloud' + href: '/cloud', + sections: [ + { + title: 'Troubleshooting', + href: '/cloud/troubleshooting' + } + ] }, { title: 'Self-hosted', From 1b104243ec7ea39c83148380a2c69ad85d0e4684 Mon Sep 17 00:00:00 2001 From: Enrique Gonzalez Date: Fri, 9 Oct 2026 20:59:29 +0000 Subject: [PATCH 4/4] docs: Link troubleshooting pages to the current support portal URL Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-726be5c1-be6e-4eaf-9008-a0e55c66fed4 --- docs/cloud/troubleshooting.mdx | 2 +- docs/self-hosted/observability/troubleshooting.mdx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/cloud/troubleshooting.mdx b/docs/cloud/troubleshooting.mdx index 539683dac..9607f3028 100644 --- a/docs/cloud/troubleshooting.mdx +++ b/docs/cloud/troubleshooting.mdx @@ -5,7 +5,7 @@ performance and health. Site admins on Cloud instances do not have access to Gra Jaeger, or server logs; Sourcegraph investigates those on your behalf. To report an error or performance issue, collect what you can from the sections below and -[contact Sourcegraph Support](https://help.sourcegraph.com/hc/en-us/requests/new) or email +[contact Sourcegraph Support](https://help.sourcegraph.com/inbox/new) or email `support@sourcegraph.com`. Don't delay contacting support if some of this information is unavailable. See [Support and SLAs](/sla#sourcegraph-cloud-sla-managed-instance) for response times. diff --git a/docs/self-hosted/observability/troubleshooting.mdx b/docs/self-hosted/observability/troubleshooting.mdx index 5ab27090d..3e102a7fe 100644 --- a/docs/self-hosted/observability/troubleshooting.mdx +++ b/docs/self-hosted/observability/troubleshooting.mdx @@ -14,7 +14,7 @@ issue or generating a high-quality issue report. any do, follow the instructions in the subsection. 1. Scan through [General scenarios](#general-scenarios) to find which scenario(s) applies to you. If any do, follow the instructions to update your instance or collect information for the issue report. -1. If you cannot resolve the issue on your own, [contact us](https://help.sourcegraph.com/hc/en-us/requests/new) with the collected +1. If you cannot resolve the issue on your own, [contact us](https://help.sourcegraph.com/inbox/new) with the collected information. Enterprise customers may alternatively file a support ticket or email `support@sourcegraph.com`.