# sync-engine

**Repository Path**: xiaote-qi/sync-engine

## Basic Information

- **Project Name**: sync-engine
- **Description**: sync-engine同步引擎，通过cannal监听mysql binlog进行解析，进行同步搜索引擎(es或mongdb等)，实现业务与同步解耦
- **Primary Language**: Java
- **License**: MulanPSL-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 3
- **Forks**: 0
- **Created**: 2023-07-02
- **Last Updated**: 2025-12-04

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# sync-engine

#### 介绍
sync-engine同步引擎，通过cannal监听mysql binlog进行解析，进行同步搜索引擎(es或mongdb等)，实现业务与同步解耦

#### 软件架构
软件架构说明
通过springboot集成canal(架构图后续补充)


#### 安装教程

####配置测试数据库，开启binlog：
```
log-bin=mysql-bin #添加这一行就ok
binlog-format=ROW #选择row模式，虽然Canal支持各种模式，但是想用otter，必须用ROW模式
server_id=1 #配置mysql replaction需要定义，不能和canal的slaveId重复
```
####下载安装canal.deployer-1.1.6.tar.gz
####配置conf/example/instance.properties:
```
#####这个id不能和目标源数据库的id一样
canal.instance.mysql.slaveId = 1234

####数据库地址，binlog订阅开始点
canal.instance.master.address = 127.0.0.1:3308
canal.instance.master.journal.name = mysql-binlog.000001
canal.instance.master.position = 16982
canal.instance.master.timestamp =

#### 配置备用源数据库
#### username/password
canal.instance.dbUsername = canal
canal.instance.dbPassword = canal
canal.instance.defaultDatabaseName =
canal.instance.connectionCharset = UTF-8

#### 订阅哪些表的binlog，支持正则表达式
canal.instance.filter.regex = .*\\..*
```

####配置conf/canal.properties:
```
canal.id= 1001
canal.ip= 127.0.0.1
canal.port= 11111
####canal通过zk做负载均衡
canal.zkServers=
####flush data to zk
canal.zookeeper.flush.period = 1000
####flush meta cursor/parse position to file
canal.file.data.dir = ${canal.conf.dir}
canal.file.flush.period = 1000
####memory store RingBuffer size, should be Math.pow(2,n)
canal.instance.memory.buffer.size = 16384
####memory store RingBuffer used memory unit size , default 1kb
canal.instance.memory.buffer.memunit = 1024
####meory store gets mode used MEMSIZE or ITEMSIZE
canal.instance.memory.batch.mode = MEMSIZE

####detecing config
canal.instance.detecting.enable = false
####canal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now()
canal.instance.detecting.sql = select 1
canal.instance.detecting.interval.time = 3
canal.instance.detecting.retry.threshold = 3
canal.instance.detecting.heartbeatHaEnable = false

####support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery
canal.instance.transaction.size =  1024
####mysql fallback connected to new master should fallback times
canal.instance.fallbackIntervalInSeconds = 60

####network config
canal.instance.network.receiveBufferSize = 16384
canal.instance.network.sendBufferSize = 16384
canal.instance.network.soTimeout = 30

####binlog filter config
canal.instance.filter.query.dcl = false
canal.instance.filter.query.dml = false
canal.instance.filter.query.ddl = false
canal.instance.filter.table.error = false

####binlog format/image check
canal.instance.binlog.format = ROW,STATEMENT,MIXED
canal.instance.binlog.image = FULL,MINIMAL,NOBLOB

####binlog ddl isolation
canal.instance.get.ddl.isolation = false

#################################################
######### destinations ############# 
#################################################
canal.destinations= example
####conf root dir
canal.conf.dir = ../conf
####auto scan instance dir add/remove and start/stop instance
canal.auto.scan = true
canal.auto.scan.interval = 5

canal.instance.global.mode = spring
canal.instance.global.lazy = false
####canal.instance.global.spring.xml = classpath:spring/memory-instance.xml
canal.instance.global.spring.xml = classpath:spring/file-instance.xml
####canal.instance.global.spring.xml = classpath:spring/default-instance.xml
```

####启动canal
```
./bin/startup.sh
```