以写数据库的方式同步 binlog by wenbobuaa · Pull Request #7 · bsc-s2/mysql-devops

wenbobuaa · 2018-11-08T12:54:29Z

和 #5 是一个事儿，那个评论有点多。改动挺大，新开了PR方便看。

从一机器读取 mysql binlog 数据向另一数据库执行 sql 语句进行 binlog 同步。

使用 go 实现，依赖的 go-mysql 包比较大，目前没有在这个目录。

详细描述： #2

wenbobuaa · 2018-11-09T03:36:26Z

go/src/app/binlogsync/binlog_sync.go

+	var errTypes = make(map[string]int)
+
+	var start = time.Now()
+	for outStat := range bs.CountCh {


这里的统计没有保证先读的 event 先被收集到，记录的binlog位置会有问题，需要修改..

drmingdrmer · 2018-11-09T03:18:58Z

go/src/app/binlogsync/binlog_sync.go

+	event    *replication.BinlogEvent
+	before   map[string]interface{}
+	after    map[string]interface{}
+	dbInfo   *DBInfo


这个dbInfo指的是源的还是目标的

这里是目标的

名字上明确下?

drmingdrmer · 2018-11-09T03:20:33Z

go/src/app/binlogsync/binlog_sync.go

+type OutMessage struct {
+	Synced   int64          `yaml:"Synced"`
+	Faild    int64          `yaml:"Faild"`
+	Rate     float64        `yaml:"Rate(rows/s)"`


直接用RowPerSec?

额...rows per sec

额...需要加复数吗?...

drmingdrmer · 2018-11-09T03:22:03Z

go/src/app/binlogsync/util.go

+	lenA := len(a)
+	lenB := len(b)
+	if lenA != lenB {
+		return 0, errors.New(fmt.Sprintf("length of slices not equals: %d != %d", lenA, lenB))


脱离这个场景, 类似python里, slice长度不同是可以比较的

drmingdrmer · 2018-11-09T03:23:52Z

go/src/app/binlogsync/util.go

+		var rst int
+		switch v.(type) {
+		case int:
+			ai := reflect.ValueOf(v).Int()


这里为啥不直接v.(int) reflect好像挺慢啊...

这里 v.(type) case 到的 int 包括了 int32 int64 等，但是 v.(int) 只能转化 v 是 int 类型的，不能转化 v 是 int64 这种类型的.. 所以才想用 reflect 都会变成 int64

那...为啥不分布case int 和int64...

这样 case 太多，还有一点是这样分 case 的话 a[i].(int) ，b[i].(int64) 不在一个 case 还要其他步骤找出来比，已知问题 yaml 把interface 类型的数字读成 int 类型，binlog 里的数字是 int64... 所以用了reflect 看起来整齐点...

reflect可能会影响性能...

case可以同时捕捉int和int64吗? 都转int64咋样...

drmingdrmer · 2018-11-09T03:26:06Z

go/src/app/binlogsync/util.go

+	}
+}
+
+func parseYAML(filename string, v interface{}) error {


要不就符合go风格, marshal unmarshal, 或者顺着py风格, load dump

嗯叫 marshal 吧

drmingdrmer · 2018-11-09T03:31:50Z

go/src/app/binlogsync/binlog_sync.go

+		if rst <= 0 {
+			return true
+		}
+		return false


直接return rst <= 0 好不...

drmingdrmer · 2018-11-09T03:45:36Z

go/src/app/binlogsync/binlog_sync.go

+}
+
+func (bs *BinlogSyncer) formatRow(srcRow []interface{}) map[string]interface{} {
+	// the first column `id` should not put in new rowValue


为啥should not...解释下...我觉得应该放进去啊...

想的是这个 id 是自增的一列，写语句里不加这行让目标表自增这列

binlog复制应该完整复制数据. id只有在咱们的场景里是不用的. 其他地方可能是有逻辑依赖的.

drmingdrmer · 2018-11-09T03:51:37Z

go/src/app/binlogsync/binlog_sync.go

+
+		bs.mutex.Lock()
+		_, err = ev.dbInfo.Conn.Execute(sql)
+		bs.mutex.Unlock()


这里为啥加锁?

这里的Conn 是几个线程公用的，Execute 实现里面有一个 setSequence 的操作会修改 Conn 的值, 测试出错了看到的

连接池考虑下吗先生?

远程调用加锁太慢了吧...

嗯考虑一下连接池

drmingdrmer · 2018-11-09T03:52:48Z

go/src/app/binlogsync/sqlUtil.go

+	whereClause = makeWhereClause(idxField, idxValue)
+	limitClause = "LIMIT 1"
+
+	return fmt.Sprintf("UPDATE %s SET %s%s%s;", tableClause, setClause, whereClause, limitClause)


恩...update 如果update的是index fields里的内容这里能处理吗...

能处理的

但是.源库的update不一定是根据index来update的吧...可能只根据primary key的id来update...

update语句在apply binlog时应该是必须用1个delete加1个insert才能完整复现update的逻辑.

况且, 考虑binlog重放, 第2次update这个东西可能会失败.

drmingdrmer

还没有记录GTID的逻辑吧?

drmingdrmer · 2018-11-13T11:47:05Z

go/src/app/binlogsync/binlog_sync.go

+
+	bs.mutex.Lock()
+	conn := bs.DBPool[shard.DBPort]
+	if conn == nil {


这个是连接池的实现吗? 没看到额

sejust

1

drmingdrmer · 2025-05-21T03:03:27Z

6

wenbobuaa self-assigned this Nov 8, 2018

wenbobuaa requested review from drmingdrmer, sejust and templexxx November 8, 2018 12:54

wenbobuaa commented Nov 9, 2018

View reviewed changes

drmingdrmer reviewed Nov 9, 2018

View reviewed changes

wenbobuaa force-pushed the binlog-sync branch from 2cd6ba3 to 97cf753 Compare November 13, 2018 08:42

drmingdrmer reviewed Nov 13, 2018

View reviewed changes

wenbobuaa added 3 commits December 5, 2018 11:57

add a binlog syncer: read binlog and write to db

31a9eba

fix database connection pool

e9b65a0

add controller

1f06031

wenbobuaa force-pushed the binlog-sync branch from 97cf753 to 1f06031 Compare December 5, 2018 04:00

sejust reviewed May 21, 2025

View reviewed changes

Conversation

wenbobuaa commented Nov 8, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

drmingdrmer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sejust left a comment

Choose a reason for hiding this comment

Uh oh!

drmingdrmer commented May 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments