Skip to content

Commit cb92510

Browse files
authored
Downgrade db schema via Curio Toolkit (#819)
* revert db * better logging * revert-feature doc * test, toolbox, nicenames * better error logging * new sql, revert to downgrade * odd AI garbage * require backups
1 parent eea7e89 commit cb92510

21 files changed

+1707
-1124
lines changed

cmd/curio/internal/translations/catalog.go

Lines changed: 1102 additions & 1089 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

cmd/curio/internal/translations/locales/en/out.gotext.json

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,27 @@
9292
"translatorComment": "Copied from source.",
9393
"fuzzy": true
9494
},
95+
{
96+
"id": "Downgrade a cluster's daatabase to a previous software version.",
97+
"message": "Downgrade a cluster's daatabase to a previous software version.",
98+
"translation": "Downgrade a cluster's daatabase to a previous software version.",
99+
"translatorComment": "Copied from source.",
100+
"fuzzy": true
101+
},
102+
{
103+
"id": "If, however, the upgrade has a serious bug and you need to downgrade, first shutdown all nodes in your cluster and then run this command. Finally, only start downgraded nodes.",
104+
"message": "If, however, the upgrade has a serious bug and you need to downgrade, first shutdown all nodes in your cluster and then run this command. Finally, only start downgraded nodes.",
105+
"translation": "If, however, the upgrade has a serious bug and you need to downgrade, first shutdown all nodes in your cluster and then run this command. Finally, only start downgraded nodes.",
106+
"translatorComment": "Copied from source.",
107+
"fuzzy": true
108+
},
109+
{
110+
"id": "YYYYMMDD when your cluster had the preferred schema. Ex: 20251128",
111+
"message": "YYYYMMDD when your cluster had the preferred schema. Ex: 20251128",
112+
"translation": "YYYYMMDD when your cluster had the preferred schema. Ex: 20251128",
113+
"translatorComment": "Copied from source.",
114+
"fuzzy": true
115+
},
95116
{
96117
"id": "Manage node config by layers. The layer 'base' will always be applied at Curio start-up.",
97118
"message": "Manage node config by layers. The layer 'base' will always be applied at Curio start-up.",

cmd/curio/internal/translations/locales/ko/messages.gotext.json

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2931,6 +2931,24 @@
29312931
"translation": "PDP 지갑이 가져와졌습니다",
29322932
"message": "PDP wallet imported",
29332933
"placeholder": null
2934+
},
2935+
{
2936+
"id": "Downgrade a cluster's daatabase to a previous software version.",
2937+
"translation": "클러스터의 데이터베이스를 이전 소프트웨어 버전으로 다운그레이드합니다.",
2938+
"message": "Downgrade a cluster's daatabase to a previous software version.",
2939+
"placeholder": null
2940+
},
2941+
{
2942+
"id": "If, however, the upgrade has a serious bug and you need to downgrade, first shutdown all nodes in your cluster and then run this command. Finally, only start downgraded nodes.",
2943+
"translation": "그러나 업그레이드에 심각한 버그가 있어 다운그레이드해야 한다면, 먼저 클러스터의 모든 노드를 종료한 뒤 이 명령을 실행하십시오. 마지막으로 다운그레이드된 노드만 다시 시작하십시오.",
2944+
"message": "If, however, the upgrade has a serious bug and you need to downgrade, first shutdown all nodes in your cluster and then run this command. Finally, only start downgraded nodes.",
2945+
"placeholder": null
2946+
},
2947+
{
2948+
"id": "YYYYMMDD when your cluster had the preferred schema. Ex: 20251128",
2949+
"translation": "클러스터에 원하는 스키마가 적용되어 있던 YYYYMMDD 날짜입니다. 예: 20251128",
2950+
"message": "YYYYMMDD when your cluster had the preferred schema. Ex: 20251128",
2951+
"placeholder": null
29342952
}
29352953
]
29362954
}

cmd/curio/internal/translations/locales/zh/messages.gotext.json

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2901,6 +2901,24 @@
29012901
"translation": "1. 使用以下命令测试您的 PDP 服务:pdptool ping --service-url https://your-domain.com --service-name public",
29022902
"message": "1. Test your PDP service with: pdptool ping --service-url https://your-domain.com --service-name public",
29032903
"placeholder": null
2904+
},
2905+
{
2906+
"id": "Downgrade a cluster's daatabase to a previous software version.",
2907+
"translation": "将集群的数据库降级到之前的软件版本。",
2908+
"message": "Downgrade a cluster's daatabase to a previous software version.",
2909+
"placeholder": null
2910+
},
2911+
{
2912+
"id": "If, however, the upgrade has a serious bug and you need to downgrade, first shutdown all nodes in your cluster and then run this command. Finally, only start downgraded nodes.",
2913+
"translation": "但是,如果升级存在严重缺陷需要降级,请先关闭集群中的所有节点,然后运行此命令。最后,仅启动已降级的节点。",
2914+
"message": "If, however, the upgrade has a serious bug and you need to downgrade, first shutdown all nodes in your cluster and then run this command. Finally, only start downgraded nodes.",
2915+
"placeholder": null
2916+
},
2917+
{
2918+
"id": "YYYYMMDD when your cluster had the preferred schema. Ex: 20251128",
2919+
"translation": "集群拥有首选架构的 YYYYMMDD 日期。例如:20251128",
2920+
"message": "YYYYMMDD when your cluster had the preferred schema. Ex: 20251128",
2921+
"placeholder": null
29042922
}
29052923
]
29062924
}

cmd/curio/toolbox.go

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,13 @@ import (
88
"net/http"
99
"net/url"
1010
"strconv"
11+
"strings"
1112

1213
"github.com/docker/go-units"
1314
"github.com/ethereum/go-ethereum/common"
15+
"github.com/fatih/color"
1416
"github.com/ipfs/go-cid"
17+
"github.com/manifoldco/promptui"
1518
"github.com/urfave/cli/v2"
1619
"golang.org/x/xerrors"
1720

@@ -36,6 +39,7 @@ var toolboxCmd = &cli.Command{
3639
Subcommands: []*cli.Command{
3740
fixMsgCmd,
3841
registerPDPServiceProviderCmd,
42+
downgradeCmd,
3943
},
4044
}
4145

@@ -439,3 +443,55 @@ var registerPDPServiceProviderCmd = &cli.Command{
439443
return nil
440444
},
441445
}
446+
447+
var downgradeCmd = &cli.Command{
448+
Name: "downgrade",
449+
Usage: translations.T("Downgrade a cluster's daatabase to a previous software version."),
450+
Description: translations.T("If, however, the upgrade has a serious bug and you need to downgrade, first shutdown all nodes in your cluster and then run this command. Finally, only start downgraded nodes."),
451+
Flags: []cli.Flag{
452+
&cli.IntFlag{
453+
Name: "last_good_date",
454+
Usage: translations.T("YYYYMMDD when your cluster had the preferred schema. Ex: 20251128"),
455+
Required: true,
456+
},
457+
},
458+
Action: func(cctx *cli.Context) error {
459+
db, err := deps.MakeDB(cctx)
460+
if err != nil {
461+
return err
462+
}
463+
464+
var runningMachines []string
465+
if err := db.Select(cctx.Context, &runningMachines, `SELECT host_and_port FROM harmony_machines
466+
WHERE last_contact > CURRENT_TIMESTAMP - INTERVAL '1 MINUTE' `); err != nil {
467+
return err
468+
}
469+
470+
if len(runningMachines) > 0 {
471+
return xerrors.Errorf("All machines must be shutdown before downgrading. Machines seen running in the past 60 seconds: %s", strings.Join(runningMachines, ", "))
472+
}
473+
474+
// Prompt user to confirm they have a database backup
475+
fmt.Println()
476+
fmt.Printf("%s Before proceeding, ensure you have a database backup.\n", color.YellowString("WARNING:"))
477+
fmt.Printf(" See: %s\n", color.CyanString("https://docs.curiostorage.org/administration/yugabyte-backup"))
478+
fmt.Println()
479+
480+
i, _, err := (&promptui.Select{
481+
Label: "Do you have a database backup?",
482+
Items: []string{
483+
"No, abort downgrade",
484+
"Yes, I have a backup",
485+
},
486+
}).Run()
487+
if err != nil {
488+
return xerrors.Errorf("selection failed: %w", err)
489+
}
490+
if i == 0 {
491+
fmt.Println("Downgrade cancelled. Please create a database backup before proceeding.")
492+
return nil
493+
}
494+
495+
return db.DowngradeTo(cctx.Context, cctx.Int("last_good_date"))
496+
},
497+
}

documentation/en/SUMMARY.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,8 @@
3737
* [Curio GUI](curio-gui.md)
3838
* [Garbage Collection](garbage-collection.md)
3939
* [Best Practices](best-practices.md)
40+
* [Administration](administration/README.md)
41+
* [YugabyteDB Backup](administration/yugabyte-backup.md)
4042
* [Logging](logging.md)
4143
* [Curio CLI](curio-cli/README.md)
4244
* [Curio](curio-cli/curio.md)
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
description: Administration guides for managing your Curio cluster
3+
---
4+
5+
# Administration
6+
7+
This section covers administrative tasks for maintaining and managing your Curio cluster.
8+
9+
## Guides
10+
11+
* [YugabyteDB Backup](yugabyte-backup.md) - How to backup and restore your database
Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
---
2+
description: How to backup and restore your YugabyteDB database for Curio
3+
---
4+
5+
# YugabyteDB Backup
6+
7+
Maintaining regular backups of your YugabyteDB database is critical for disaster recovery and before performing operations like software downgrades. This guide covers the essential backup and restore procedures for your Curio cluster's database.
8+
9+
{% hint style="danger" %}
10+
**Always create a backup before running `curio toolbox downgrade`** or performing any major cluster operations. Database schema changes during upgrades may not be reversible without a backup.
11+
{% endhint %}
12+
13+
## Prerequisites
14+
15+
- Access to your YugabyteDB cluster
16+
- The `ysql_dump` and `ysqlsh` utilities (included with YugabyteDB installation)
17+
- Sufficient disk space for backup files
18+
19+
## Backup Methods
20+
21+
### Method 1: Using ysql_dump (Recommended)
22+
23+
The `ysql_dump` utility creates a logical backup of your database that can be restored to any YugabyteDB cluster.
24+
25+
#### Full Database Backup
26+
27+
```bash
28+
ysql_dump -h <yugabyte-host> -p 5433 -U <username> -d <database> -F c -f curio_backup_$(date +%Y%m%d_%H%M%S).dump
29+
```
30+
31+
**Parameters:**
32+
- `-h`: YugabyteDB host address
33+
- `-p`: YSQL port (default: 5433)
34+
- `-U`: Database username
35+
- `-d`: Database name (typically `curio` or your configured database name)
36+
- `-F c`: Custom format (compressed, supports parallel restore)
37+
- `-f`: Output filename
38+
39+
#### Example with typical Curio configuration
40+
41+
```bash
42+
ysql_dump -h 127.0.0.1 -p 5433 -U yugabyte -d curio -F c -f curio_backup_$(date +%Y%m%d_%H%M%S).dump
43+
```
44+
45+
#### Schema-Only Backup
46+
47+
To backup only the database schema without data:
48+
49+
```bash
50+
ysql_dump -h <yugabyte-host> -p 5433 -U <username> -d <database> --schema-only -f curio_schema_$(date +%Y%m%d_%H%M%S).sql
51+
```
52+
53+
### Method 2: Using ysqlsh with COPY
54+
55+
For smaller databases or specific tables:
56+
57+
```bash
58+
ysqlsh -h <yugabyte-host> -p 5433 -U <username> -d <database> -c "\COPY <table_name> TO 'table_backup.csv' WITH CSV HEADER"
59+
```
60+
61+
## Restore Procedures
62+
63+
### Restore from ysql_dump backup
64+
65+
#### Full Restore
66+
67+
{% hint style="warning" %}
68+
Ensure all Curio nodes are **stopped** before restoring a backup.
69+
{% endhint %}
70+
71+
```bash
72+
# First, drop and recreate the database if needed
73+
ysqlsh -h <yugabyte-host> -p 5433 -U <username> -c "DROP DATABASE IF EXISTS curio;"
74+
ysqlsh -h <yugabyte-host> -p 5433 -U <username> -c "CREATE DATABASE curio;"
75+
76+
# Restore from backup
77+
pg_restore -h <yugabyte-host> -p 5433 -U <username> -d curio -F c curio_backup_YYYYMMDD_HHMMSS.dump
78+
```
79+
80+
#### Restore from SQL dump
81+
82+
```bash
83+
ysqlsh -h <yugabyte-host> -p 5433 -U <username> -d curio -f curio_backup.sql
84+
```
85+
86+
## Automated Backup Script
87+
88+
Create a backup script for regular automated backups:
89+
90+
```bash
91+
#!/bin/bash
92+
# curio-db-backup.sh
93+
94+
BACKUP_DIR="/path/to/backups"
95+
YB_HOST="127.0.0.1"
96+
YB_PORT="5433"
97+
YB_USER="yugabyte"
98+
YB_DB="curio"
99+
RETENTION_DAYS=7
100+
101+
# Create backup directory if it doesn't exist
102+
mkdir -p "$BACKUP_DIR"
103+
104+
# Create backup
105+
BACKUP_FILE="$BACKUP_DIR/curio_backup_$(date +%Y%m%d_%H%M%S).dump"
106+
ysql_dump -h "$YB_HOST" -p "$YB_PORT" -U "$YB_USER" -d "$YB_DB" -F c -f "$BACKUP_FILE"
107+
108+
if [ $? -eq 0 ]; then
109+
echo "Backup created successfully: $BACKUP_FILE"
110+
111+
# Remove backups older than retention period
112+
find "$BACKUP_DIR" -name "curio_backup_*.dump" -mtime +$RETENTION_DAYS -delete
113+
echo "Cleaned up backups older than $RETENTION_DAYS days"
114+
else
115+
echo "Backup failed!"
116+
exit 1
117+
fi
118+
```
119+
120+
Make the script executable and add it to cron:
121+
122+
```bash
123+
chmod +x curio-db-backup.sh
124+
125+
# Add to crontab for daily backups at 2 AM
126+
crontab -e
127+
# Add: 0 2 * * * /path/to/curio-db-backup.sh >> /var/log/curio-backup.log 2>&1
128+
```
129+
130+
## Best Practices
131+
132+
1. **Regular Backups**: Schedule automated daily backups, especially for production clusters
133+
2. **Test Restores**: Periodically verify your backups by performing test restores
134+
3. **Off-site Storage**: Store backup copies in a different location or cloud storage
135+
4. **Pre-upgrade Backups**: Always create a fresh backup before upgrading or downgrading Curio
136+
5. **Monitor Backup Size**: Track backup sizes to ensure adequate storage capacity
137+
138+
## Troubleshooting
139+
140+
### Connection Issues
141+
142+
If you encounter connection errors:
143+
144+
```bash
145+
# Verify YugabyteDB is running
146+
yugabyted status
147+
148+
# Check connectivity
149+
ysqlsh -h <host> -p 5433 -U yugabyte -c "SELECT version();"
150+
```
151+
152+
### Permission Errors
153+
154+
Ensure your database user has sufficient privileges:
155+
156+
```sql
157+
GRANT ALL PRIVILEGES ON DATABASE curio TO <username>;
158+
```
159+
160+
## Additional Resources
161+
162+
- [YugabyteDB Backup and Restore Documentation](https://docs.yugabyte.com/preview/manage/backup-restore/)
163+
- [ysql_dump Reference](https://docs.yugabyte.com/preview/admin/ysql-dump/)
164+
- [YugabyteDB Best Practices](https://docs.yugabyte.com/preview/develop/best-practices-ysql/)

documentation/en/curio-cli/curio.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1296,6 +1296,7 @@ USAGE:
12961296
COMMANDS:
12971297
fix-msg Updated DB with message data missing from chain node
12981298
register-pdp-service-provider Register a PDP service provider with Filecoin Service Registry Contract
1299+
downgrade Downgrade a cluster's daatabase to a previous software version.
12991300
help, h Shows a list of commands or help for one command
13001301
13011302
OPTIONS:
@@ -1337,3 +1338,19 @@ OPTIONS:
13371338
--token-address value Token contract for payment (IERC20(address(0)) for FIL)
13381339
--help, -h show help
13391340
```
1341+
1342+
### curio toolbox downgrade
1343+
```
1344+
NAME:
1345+
curio toolbox downgrade - Downgrade a cluster's daatabase to a previous software version.
1346+
1347+
USAGE:
1348+
curio toolbox downgrade [command options]
1349+
1350+
DESCRIPTION:
1351+
If, however, the upgrade has a serious bug and you need to downgrade, first shutdown all nodes in your cluster and then run this command. Finally, only start downgraded nodes.
1352+
1353+
OPTIONS:
1354+
--last_good_date value YYYYMMDD when your cluster had the preferred schema. Ex: 20251128 (default: 0)
1355+
--help, -h show help
1356+
```

documentation/en/versions.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,11 @@ Configurations and the number of machines needed: A: Lotus, Curio (numerous), Yu
3636
* Curio's DEBs include curio-cuda (for Nvidia) and curio-opencl (others like ATI).
3737
* These can be mixed in a Curio cluster as they only relate to the hardware on the box.
3838

39+
## Database Schema Versions
40+
* When the latest Curio starts-up, it applies any upgrades & migrations to Yugabyte's schema.
41+
* This may cause errors on other nodes in your cluster that run the old version (low likelihood), which has the simple solution of completing the upgrade.
42+
* If, however, the upgrade has a serious bug and you need to downgrade, "curio toolbox downgrade --last_good_date=20250515"
43+
3944
## Notes
4045

4146
* Forest (0.19+ & Docker Watchtower) is a light alternative to Lotus Client. It meets Curio's needs, but Boost compatibility is in development.

0 commit comments

Comments
 (0)