mirror of
https://github.com/supabase/supabase.git
synced 2026-05-11 02:20:29 +08:00
* feat: replace reports links with observability * feat: update with chris suggestions * Add redirect --------- Co-authored-by: Chris Chinchilla <chris.ward@supabase.io>
200 lines
6.9 KiB
Plaintext
200 lines
6.9 KiB
Plaintext
---
|
||
title = "pg_cron debugging guide"
|
||
github_url = "https://github.com/orgs/supabase/discussions/30168"
|
||
date_created = "2024-10-29T15:12:11+00:00"
|
||
topics = [ "database" ]
|
||
keywords = [ "pg_cron", "cron", "jobs", "scheduler", "debugging" ]
|
||
database_id = "9a55c946-877f-46ae-8b57-51934e02a36c"
|
||
---
|
||
|
||
This is a general guide for debugging pg_cron. Below lists issues and how to debug them
|
||
|
||
# Cannot create/edit/delete cron jobs
|
||
|
||
Cron jobs can only be modified with the respective SQL functions:
|
||
|
||
- Schedule: [cron.schedule](/docs/guides/database/extensions/pg_cron#delete-data-every-week)
|
||
- Alter: [cron.alter_job](/docs/guides/database/extensions/pg_cron#edit-a-job)
|
||
- Delete: [cron.unschedule](/docs/guides/database/extensions/pg_cron#unschedule-a-job)
|
||
|
||
If you are trying to make changes, use the cron functions. If the cron functions are inaccessible, contact [Support](/dashboard/support/new)
|
||
|
||
<br />
|
||
|
||
---
|
||
|
||
# Cron Jobs are not running
|
||
|
||
> You should consider initiating a software upgrade in the [Infrastructure Settings](/dashboard/project/_/settings/infrastructure) if your Postgres version is below v15.6.1.122. Upgrading will give you access to pg_cron v1.6.4+, which has many bug fixes and auto-revive capabilities.
|
||
|
||
## Debugging steps:
|
||
|
||
### Check to see if "pg_cron scheduler" is active
|
||
|
||
pg_cron operates as the `pg_cron scheduler` process within Postgres. Use the below query to check if the worker is active
|
||
|
||
```sql
|
||
SELECT
|
||
pid as process_id,
|
||
usename as database_user,
|
||
application_name,
|
||
backend_start as when_process_began,
|
||
wait_event_type,
|
||
state,
|
||
query,
|
||
backend_type
|
||
FROM pg_stat_activity where application_name ilike 'pg_cron scheduler';
|
||
```
|
||
|
||
If the query does not return a row, the worker has died. To revive it, you must go to the [General Settings](/dashboard/project/_/settings/general) and initiate a fast reboot:
|
||
|
||
<img
|
||
width="719"
|
||
alt="Screenshot 2024-10-29 at 12 27 24 AM"
|
||
src="https://github.com/user-attachments/assets/d7a2956c-08e0-4ebc-be08-d14a2383f36b"
|
||
/>
|
||
|
||
<br />
|
||
|
||
### Check the `cron.job_run_details` table for more information
|
||
|
||
pg_cron creates logs in its own table `cron.job_run_details"`. The below query checks for issues from the past 5 days :
|
||
|
||
```sql
|
||
SELECT *
|
||
FROM cron.job_run_details
|
||
WHERE
|
||
(status <> 'succeeded' AND status <> 'running')
|
||
AND
|
||
start_time > NOW() - INTERVAL '5 days'
|
||
ORDER BY start_time DESC
|
||
LIMIT 10;
|
||
```
|
||
|
||
Respond to the errors exposed appropriately.
|
||
|
||
> Long running jobs may show timeout errors. For jobs that are intended to execute for extended periods, consider wrapping their queries in functions with custom timeouts ([guide](/docs/guides/database/postgres/timeouts#function-level)).
|
||
|
||
<br />
|
||
|
||
### Check if there are too many cron jobs running concurrently
|
||
|
||
pg_cron supports up to 32 concurrent jobs, each using a database connection. If too many jobs are running simultaneously, space them out to prevent connection overload and job failure.
|
||
|
||
The below queries shows all pg_cron jobs:
|
||
|
||
```sql
|
||
-- All jobs
|
||
select schedule, jobname, command from cron.job;
|
||
|
||
-- Count jobs
|
||
select COUNT(*) from cron.job;
|
||
```
|
||
|
||
The below query can be used to find actively querying jobs:
|
||
|
||
```sql
|
||
SELECT
|
||
pid as process_id,
|
||
usename as database_user,
|
||
application_name,
|
||
backend_start as when_process_began,
|
||
wait_event_type,
|
||
state,
|
||
query,
|
||
backend_type
|
||
FROM pg_stat_activity where application_name ilike 'pg_cron';
|
||
```
|
||
|
||
You can view your concurrent peak connection usage throughout the day at the bottom of the [Observability Dashboard](/dashboard/project/_/observability/database)
|
||
|
||
<br />
|
||
|
||
### Check for database strain
|
||
|
||
Unfortunately, excessive resource strain can slow down or disrupt jobs.
|
||
|
||
Go to the [reports page](/dashboard/project/_/observability/database) (or [Supabase Grafana](/docs/guides/monitoring-troubleshooting/metrics#deploying-supabase-grafana) if you have it setup), and check for signs of resource exhaustion. If it's clear your database is under pressure, consider upgrading your compute add-on or following the advice from one of the optimization guides:
|
||
|
||
- [Connections](https://github.com/orgs/supabase/discussions/27141)
|
||
- [Disk/IO](https://github.com/orgs/supabase/discussions/27003)
|
||
- [Memory](https://github.com/orgs/supabase/discussions/27021)
|
||
- [CPU](https://github.com/orgs/supabase/discussions/27022)
|
||
|
||
It is important to make sure you are running the latest release of pg_cron (1.6.4) if you're noticing strain. It is the most robust.
|
||
|
||
<br />
|
||
|
||
### Check the log explorer for more information
|
||
|
||
Although pg*cron records errors in the `cron.job_run_details` table, in rare cases, more information can be found in the general Postgres logs. You can check the [Log Explorer](/dashboard/project/*/logs/explorer) for failure events with the following query
|
||
|
||
```sql
|
||
select
|
||
cast(postgres_logs.timestamp as datetime) as timestamp,
|
||
event_message,
|
||
parsed.error_severity,
|
||
parsed.user_name,
|
||
parsed.query,
|
||
parsed.detail,
|
||
parsed.hint,
|
||
parsed.sql_state_code,
|
||
parsed.backend_type,
|
||
parsed.application_name
|
||
from
|
||
postgres_logs
|
||
cross join unnest(metadata) as metadata
|
||
cross join unnest(metadata.parsed) as parsed
|
||
where
|
||
regexp_contains(parsed.error_severity, 'ERROR|FATAL|PANIC')
|
||
and regexp_contains(parsed.application_name, 'pg_cron')
|
||
order by timestamp desc
|
||
limit 100;
|
||
```
|
||
|
||
If you're interested in modifying the query, there is an advanced [guide](https://github.com/orgs/supabase/discussions/26224) for navigating the Postgres logs and a general purpose [one](/docs/guides/platform/advanced-log-filtering) for applying filters.
|
||
|
||
<br />
|
||
|
||
### Create custom logs within cron jobs
|
||
|
||
If it's still not clear what is occurring you may be able to capture more logs by running the pg_cron query inside a database function:
|
||
|
||
```sql
|
||
create or replace function log_example()
|
||
returns void
|
||
language plpgsql
|
||
as $$
|
||
begin
|
||
|
||
-- Logging start of function
|
||
raise log 'logging start of cron function call: (%)', (select now());
|
||
|
||
-- INSERT LOGIC HERE
|
||
|
||
-- Logging end of function
|
||
raise log 'logging end of cron function call: (%)', (select now());
|
||
|
||
exception
|
||
-- Handle exceptions here if needed
|
||
when others then
|
||
raise exception 'An error occurred in cron function <insert name here>. ERROR MESSAGE: %', sqlerrm;
|
||
|
||
end;
|
||
$$;
|
||
```
|
||
|
||
You can then search for your custom messages in the [Logs Interface](/dashboard/project/_/logs/postgres-logs)
|
||
|
||
<br />
|
||
|
||
### Upgrading pg_cron version
|
||
|
||
The current version of pg*cron on Supabase is 1.6.4. It comes with a [few bug fixes](https://github.com/citusdata/pg_cron/releases/tag/v1.6.4). You should consider upgrading to Postgres v15.6.1.122+ in the[ Infrastructure Settings](/dashboard/project/*/settings/infrastructure) to get the latest extension.
|
||
|
||
<br />
|
||
|
||
### Contacting support and the maintainers
|
||
|
||
Although Supabase includes the extension, it is maintained by Citus (a Microsoft subsidiary). You can contact Support for more help, but you should also consider creating an issue in the [pg_cron repo](https://github.com/citusdata/pg_cron).
|