Problem: New validator projection. Implement migration scripts and views [Part 2] by ysong42 · Pull Request #684 · crypto-com/chain-indexing

ysong42 · 2022-02-11T09:52:27Z

Solution: fix #665 #678

Changes

Implement views and migrations for new validator projection.
Extend the example app. Add new temporary APIs, to retrieve validators, delegations, unbondingDelegations and redelegations at specific height.

New Temporary APIs Examples

# Get Validators list
http://localhost:8080/api/test/validators?height=600000

# Get Specific validator
http://localhost:8080/api/test/validators/tcrocnclcons1rxvzs3kzdekrjazlshlua2v4pz24v0hpedgha2?height=150000

# Get Validator Delegations
http://localhost:8080/api/test/validators/crocncl149dyku4d4c36dmvvw583xqcflau3d9x303ffcj/delegations?height=4234440&limit=1000

# Get Validator UnbondingDelegations
http://localhost:8080/api/test/validators/crocncl149dyku4d4c36dmvvw583xqcflau3d9x303ffcj/unbonding_delegations?height=4234440&limit=1000

# Get Validator Redelegations
http://localhost:8080/api/test/validators/tcrocncl15mstquxa00lw5499tptsrw888tkw8dkt0c8xud/redelegations?height=2619910&limit=1000

# Get Delegator Delegations
http://localhost:8080/api/test/delegators/tcro1k4nz2yalsuzuzxexcxuughp33cfqls7e6utarx/delegations?height=150000

Performance and Correctness

Please refer to: #678 (comment)

…and validators tables.

… code.

davcrypto · 2022-02-15T07:40:08Z

example/config/config.yaml

        channel_id: "channel-131"
        starting_height: 899374
+      ValidatorDelegation:
+        unbonding_time: "2419200000000000ns"


suggest to change to ms / s

oh this is from Crypto.org genesis. I just copied from the genesis.

Later when we support proposals updaing params in this projection, we will remove this config.

davcrypto · 2022-02-15T07:41:47Z

example/config/config.yaml

        starting_height: 899374
+      ValidatorDelegation:
+        unbonding_time: "2419200000000000ns"
+        slash_fraction_double_sign: "0.050000000000000000"


May i know that are those slash fractions can be change by proposal?

https://crypto.org/explorer/proposals

You can check those proposals there. So there are different type of proposals. And there are module params that could be configured through gov module's proposal.

davcrypto · 2022-02-15T07:49:34Z

infrastructure/httpapi/handlers/validator_delegation.go

+	httpapi.Success(ctx, validator)
+}
+
+func (handler *ValidatorDelegation) ListValidator(ctx *fasthttp.RequestCtx) {


may i know what is the diff between this ListValidator and the List in Validator projection view?

This one is on http handler. It is for parsing the query params. Then triggered the under-lying view logic to interact with DB.

davcrypto · 2022-02-15T07:54:24Z

projection/validator_delegation/migrations/20220117104331_create_view_vd_delegations.up.sql

@@ -0,0 +1,13 @@
+CREATE EXTENSION btree_gist;
+
+CREATE TABLE view_vd_delegations (


suggest to change it in completed form instead of vd

Updated. And the similar places below

davcrypto · 2022-02-15T07:54:35Z

projection/validator_delegation/migrations/20220117110519_create_view_vd_evidences.up.sql

@@ -0,0 +1,8 @@
+CREATE TABLE view_vd_evidences (


davcrypto · 2022-02-15T07:54:41Z

...tion/validator_delegation/migrations/20220117111405_create_view_vd_redelegation_queue.up.sql

@@ -0,0 +1,6 @@
+CREATE TABLE view_vd_redelegation_queue (


davcrypto · 2022-02-15T07:54:46Z

projection/validator_delegation/migrations/20220117111937_create_view_vd_redelegations.up.sql

@@ -0,0 +1,19 @@
+CREATE TABLE view_vd_redelegations (


davcrypto · 2022-02-15T07:54:52Z

...idator_delegation/migrations/20220117130920_create_view_vd_unbonding_delegation_queue.up.sql

@@ -0,0 +1,6 @@
+CREATE TABLE view_vd_unbonding_delegation_queue (


davcrypto · 2022-02-15T07:54:59Z

...n/validator_delegation/migrations/20220117131200_create_view_vd_unbonding_delegations.up.sql

@@ -0,0 +1,13 @@
+CREATE TABLE view_vd_unbonding_delegations (


davcrypto · 2022-02-15T07:55:06Z

...on/validator_delegation/migrations/20220117132223_create_view_vd_unbonding_validators.up.sql

@@ -0,0 +1,6 @@
+CREATE TABLE view_vd_unbonding_validators (


davcrypto · 2022-02-15T07:55:13Z

projection/validator_delegation/migrations/20220117133017_create_view_vd_validators.up.sql

@@ -0,0 +1,28 @@
+CREATE TABLE view_vd_validators (


davcrypto · 2022-02-15T08:12:12Z

projection/validator_delegation/view/delegations.go

 	return nil
 }

+func (view *DelegationsView) checkIfDelegationRecordExistByHeightLowerBound(row DelegationRow) (bool, error) {


suggest IsExistedByLowerBoundHeight(validatorAddress, delegatorAddress string, height int64) (bool, error) for readability

Thanks! Updated. I will use isExistedByLowerBoundHeight to keep the func as private.

davcrypto · 2022-02-15T08:19:54Z

projection/validator_delegation/view/delegations.go

+	return view.setDelegationRecordHeightUpperBound(row)
+}
+
+func (view *DelegationsView) setDelegationRecordHeightUpperBound(row DelegationRow) error {


UpdateUpperBoundHeight

Thanks, updated. I will use updateUpperBoundHeight, as I want to keep it as private.

davcrypto · 2022-02-15T08:26:05Z

projection/validator_delegation/view/redelegations.go

 	return nil
 }

+func (view *RedelegationsView) checkIfRedelegationRecordExistByHeightLowerBound(row RedelegationRow) (bool, error) {


suggest change toIsExistedByLowerBoundHeight(delegatorAddress, validatorSrcAddress, validatorDstAddress string, height int64) for readability

Thanks, updated.

davcrypto · 2022-02-15T08:26:55Z

projection/validator_delegation/view/redelegations.go

+	return view.setRedelegationRecordHeightUpperBound(row)
+}
+
+func (view *RedelegationsView) setRedelegationRecordHeightUpperBound(row RedelegationRow) error {


UpdateUpperBoundHeight

Thanks, updated.

davcrypto · 2022-02-15T08:28:37Z

...idator_delegation/migrations/20220117130920_create_view_vd_unbonding_delegation_queue.up.sql

+CREATE TABLE view_vd_unbonding_delegation_queue (
+    id BIGSERIAL,
+    completion_time BIGINT NOT NULL UNIQUE,
+    dv_pairs JSONB NOT NULL,


Can we add some description for short formed column?

davcrypto · 2022-02-15T08:29:40Z

...tion/validator_delegation/migrations/20220117111405_create_view_vd_redelegation_queue.up.sql

+CREATE TABLE view_vd_redelegation_queue (
+    id BIGSERIAL,
+    completion_time BIGINT NOT NULL UNIQUE,
+    dvv_triplets JSONB NOT NULL,


Can we add some description for short formed column?

davcrypto · 2022-02-15T08:31:07Z

projection/validator_delegation/view/unbonding_delegations.go

 	return nil
 }

+func (view *UnbondingDelegationsView) checkIfUnbondingDelegationRecordExistByHeightLowerBound(row UnbondingDelegationRow) (bool, error) {


similar suggestion here

Thanks, updated.

davcrypto · 2022-02-15T08:31:14Z

projection/validator_delegation/view/unbonding_delegations.go

+	return view.setUnbondingDelegationRecordHeightUpperBound(row)
+}
+
+func (view *UnbondingDelegationsView) setUnbondingDelegationRecordHeightUpperBound(row UnbondingDelegationRow) error {


similar suggestion here

Thanks, updated.

davcrypto · 2022-02-15T08:32:32Z

projection/validator_delegation/view/validators.go

 	return nil
 }

+func (view *ValidatorsView) checkIfValidatorRecordExistByHeightLowerBound(row ValidatorRow) (bool, error) {


similar suggestion here

Thanks, updated.

davcrypto · 2022-02-15T08:32:41Z

projection/validator_delegation/view/validators.go

+	return view.setValidatorRecordHeightUpperBound(row)
+}
+
+func (view *ValidatorsView) setValidatorRecordHeightUpperBound(row ValidatorRow) error {


similar suggestion here

Thanks, updated.

davcrypto · 2022-02-15T08:33:43Z

projection/validator_delegation/view/validators.go

 }

-func (view *ValidatorsView) FindByTendermintAddr(tendermintAddress string, height int64) (ValidatorRow, error) {
+func (view *ValidatorsView) findBy(


is it intended to make it private?

Yes. Users are expected to use FindByOperatorAddr and FindByConsensusNodeAddr to find a specific record.

The FindByOperatorAddr and FindByConsensusNodeAddr will invoke findBy internally. These two functions are mainly for readability. The real logic is in findBy.

FindByTendermintAddr is not used in implementation, so I simply remove it.

ysong42 · 2022-02-21T08:28:59Z

@davcrypto Thanks for the review! The code is updated! Feel free to have another review 🙏

ysong42 added 21 commits February 11, 2022 11:12

Commit migration scripts and some of the implemented views.

fe5ce23

Implemented remaining views.

1670d06

Add logic for projection prepare config

9dcbc8a

Add config init. Fixed migration script.

cd561ef

Added logic for handle MsgCreateValidator in Genesis

23000b2

Added test APIs to fetch validators and delegations

69ad576

Add temporary operation performance log DB schema.

46f686e

Add height index on delegations table

2413edf

Refactor migrations

50e375b

Fixed validator json fields

0e75837

Fix a bug that validator is not removed when its token decrease to 0.

24c890e

Update migrations script to test range type performance

826fc25

Apply range types for other tables in the new projection.

1d33b46

Apply range type to delegations, unbondingDelegations, redelegations …

c321a08

…and validators tables.

Add what HTTP API for unbondingDelegations and redelegations.

467494f

Update index and constraints for the new projection

5387cab

Fix migrations script

541bf2e

Commit latest working code. Include the performance log related debug…

886dd28

… code.

Update example app

0081eeb

Removed performance trace related code

0e0c234

Updated example

f5211d1

ysong42 requested review from calvinaco and davcrypto as code owners February 11, 2022 09:52

Changed back a example config

720b1da

davcrypto reviewed Feb 15, 2022

View reviewed changes

ysong42 added 2 commits February 21, 2022 10:44

Updated the PR according to comments.

38ae3fe

Updated example accordingly

d80fba5

ysong42 requested a review from davcrypto February 21, 2022 08:28

		@@ -0,0 +1,13 @@
		CREATE EXTENSION btree_gist;

		CREATE TABLE view_vd_delegations (

		@@ -0,0 +1,6 @@
		CREATE TABLE view_vd_unbonding_delegation_queue (

Conversation

ysong42 commented Feb 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

New Temporary APIs Examples

Performance and Correctness

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ysong42 Feb 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ysong42 Feb 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

ysong42 commented Feb 11, 2022 •

edited

Loading

ysong42 Feb 21, 2022 •

edited

Loading

ysong42 Feb 21, 2022 •

edited

Loading

ysong42 Feb 21, 2022 •

edited

Loading