升级至企业版3.3.0

调整代码结构

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,51 @@
+---
+name: 报告Bug
+about: 报告KnowStreaming的相关Bug
+title: ''
+labels: bug
+assignees: ''
+
+---
+
+- [ ] 我已经在 [issues](https://github.com/didi/KnowStreaming/issues) 搜索过相关问题了,并没有重复的。
+
+ 你是否希望来认领这个Bug。
+
+ 「   Y   /   N  」
+
+### 环境信息
+
+* KnowStreaming version :   <font size=4 color =red>   xxx </font>
+* Operating System version :  <font size=4 color =red> xxx </font>
+* Java version : <font size=4 color =red> xxx </font>
+
+
+### 重现该问题的步骤
+
+1. xxx
+   
+
+
+2. xxx
+   
+
+3. xxx
+
+
+
+### 预期结果
+
+<!-- 写下应该出现的预期结果？-->
+
+### 实际结果
+
+<!-- 实际发生了什么? -->
+
+
+---
+
+如果有异常，请附上异常Trace:
+
+```
+Just put your stack trace here!
+```

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,8 @@
+blank_issues_enabled: true
+contact_links:
+  - name: 讨论问题
+    url: https://github.com/didi/KnowStreaming/discussions/new
+    about: 发起问题、讨论 等等
+  - name: KnowStreaming官网
+    url: https://knowstreaming.com/
+    about: KnowStreaming website

.github/ISSUE_TEMPLATE/detail_optimizing.md

@@ -0,0 +1,26 @@
+---
+name: 优化建议
+about: 相关功能优化建议
+title: ''
+labels: Optimization Suggestions
+assignees: ''
+
+---
+
+- [ ] 我已经在 [issues](https://github.com/didi/KnowStreaming/issues) 搜索过相关问题了,并没有重复的。
+
+ 你是否希望来认领这个优化建议。
+
+ 「   Y   /   N  」
+
+### 环境信息
+
+* KnowStreaming version :   <font size=4 color =red>   xxx </font>
+* Operating System version :  <font size=4 color =red> xxx </font>
+* Java version : <font size=4 color =red> xxx </font>
+
+### 需要优化的功能点
+
+
+### 建议如何优化
+

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: 提议新功能/需求
+about: 给KnowStreaming提一个功能需求
+title: ''
+labels: feature
+assignees: ''
+
+---
+
+- [ ] 我在 [issues](https://github.com/didi/KnowStreaming/issues) 中并未搜索到与此相关的功能需求。
+- [ ] 我在 [release note](https://github.com/didi/KnowStreaming/releases) 已经发布的版本中并没有搜到相关功能.
+
+你是否希望来认领这个Feature。
+
+「   Y   /   N  」
+
+
+## 这里描述需求
+<!--请尽可能的描述清楚您的需求 -->
+

.github/ISSUE_TEMPLATE/question.md

@@ -0,0 +1,12 @@
+---
+name: 提个问题
+about: 问KnowStreaming相关问题
+title: ''
+labels: question
+assignees: ''
+
+---
+
+- [ ] 我已经在 [issues](https://github.com/didi/KnowStreaming/issues) 搜索过相关问题了,并没有重复的。
+
+## 在这里提出你的问题

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,23 @@
+请不要在没有先创建Issue的情况下创建Pull Request。
+
+## 变更的目的是什么
+
+XXXXX
+
+## 简短的更新日志
+
+XX
+
+## 验证这一变化
+
+XXXX
+
+请遵循此清单，以帮助我们快速轻松地整合您的贡献：
+
+* [ ] 一个 PR（Pull Request的简写）只解决一个问题，禁止一个 PR 解决多个问题；
+* [ ] 确保 PR 有对应的 Issue（通常在您开始处理之前创建），除非是书写错误之类的琐碎更改不需要 Issue ；
+* [ ] 格式化 PR 及 Commit-Log 的标题及内容，例如 #861 。PS：Commit-Log 需要在 Git Commit 代码时进行填写，在 GitHub 上修改不了；
+* [ ] 编写足够详细的 PR 描述，以了解 PR 的作用、方式和原因；
+* [ ] 编写必要的单元测试来验证您的逻辑更正。如果提交了新功能或重大更改，请记住在 test 模块中添加 integration-test；
+* [ ] 确保编译通过，集成测试通过；
+

.gitignore

@@ -109,4 +109,8 @@ out/*
 dist/
 dist/*
 km-rest/src/main/resources/templates/
-*dependency-reduced-pom*
+*dependency-reduced-pom*
+#filter flattened xml
+*/.flattened-pom.xml
+.flattened-pom.xml
+*/*/.flattened-pom.xml

CODE_OF_CONDUCT.md

@@ -0,0 +1,74 @@
+
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project, and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of experience,
+education, socio-economic status, nationality, personal appearance, race,
+religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+  advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at  https://knowstreaming.com/support-center .  All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org

CONTRIBUTING.md

@@ -1,28 +1,150 @@
-# Contribution Guideline
 
-Thanks for considering to contribute this project. All issues and pull requests are highly appreciated.
 
-## Pull Requests
 
-Before sending pull request to this project, please read and follow guidelines below.
+# 为KnowStreaming做贡献
 
-1. Branch: We only accept pull request on `dev` branch.
-2. Coding style: Follow the coding style used in LogiKM.
-3. Commit message: Use English and be aware of your spell.
-4. Test: Make sure to test your code.
 
-Add device mode, API version, related log, screenshots and other related information in your pull request if possible.
+欢迎👏🏻来到KnowStreaming！本文档是关于如何为KnowStreaming做出贡献的指南。
 
-NOTE: We assume all your contribution can be licensed under the [Apache License 2.0](LICENSE).
+如果您发现不正确或遗漏的内容, 请留下意见/建议。
 
-## Issues
+## 行为守则
+请务必阅读并遵守我们的 [行为准则](./CODE_OF_CONDUCT.md).
 
-We love clearly described issues. :)
 
-Following information can help us to resolve the issue faster.
 
-* Device mode and hardware information.
+## 贡献
-* API version.
+
-* Logs.
+**KnowStreaming** 欢迎任何角色的新参与者，包括 **User** 、**Contributor**、**Committer**、**PMC** 。
-* Screenshots.
+
-* Steps to reproduce the issue.
+我们鼓励新人积极加入 **KnowStreaming** 项目，从User到Contributor、Committer ，甚至是 PMC 角色。
+
+为了做到这一点，新人需要积极地为 **KnowStreaming** 项目做出贡献。以下介绍如何对 **KnowStreaming** 进行贡献。
+
+
+### 创建/打开  Issue
+
+如果您在文档中发现拼写错误、在代码中**发现错误**或想要**新功能**或想要**提供建议**，您可以在 GitHub 上[创建一个Issue](https://github.com/didi/KnowStreaming/issues/new/choose) 进行报告。
+
+
+如果您想直接贡献, 您可以选择下面标签的问题。
+
+- [contribution welcome](https://github.com/didi/KnowStreaming/labels/contribution%20welcome) : 非常需要解决/新增  的Issues
+- [good first issue](https://github.com/didi/KnowStreaming/labels/good%20first%20issue): 对新人比较友好, 新人可以拿这个Issue来练练手热热身。
+
+<font color=red ><b> 请注意，任何 PR 都必须与有效issue相关联。否则，PR 将被拒绝。</b></font>
+
+
+
+### 开始你的贡献
+
+**分支介绍**
+
+我们将 `dev`分支作为开发分支, 说明这是一个不稳定的分支。
+
+此外,我们的分支模型符合 [https://nvie.com/posts/a-successful-git-branching-model/](https://nvie.com/posts/a-successful-git-branching-model/). 我们强烈建议新人在创建PR之前先阅读上述文章。
+
+
+
+**贡献流程**
+
+为方便描述,我们这里定义一下2个名词：
+
+自己Fork出来的仓库是私人仓库, 我们这里称之为 ：**分叉仓库**
+Fork的源项目,我们称之为：**源仓库**
+
+
+现在，如果您准备好创建PR, 以下是贡献者的工作流程:
+
+1. Fork [KnowStreaming](https://github.com/didi/KnowStreaming) 项目到自己的仓库
+
+2. 从源仓库的`dev`拉取并创建自己的本地分支,例如: `dev`
+3. 在本地分支上对代码进行修改
+4. Rebase 开发分支, 并解决冲突
+5. commit 并 push 您的更改到您自己的**分叉仓库**
+6. 创建一个 Pull Request 到**源仓库**的`dev`分支中。
+7. 等待回复。如果回复的慢，请无情的催促。
+
+
+更为详细的贡献流程请看：[贡献流程](./docs/contributer_guide/贡献流程.md)
+
+创建Pull Request时：
+
+1. 请遵循 PR的 [模板](./.github/PULL_REQUEST_TEMPLATE.md)
+2. 请确保 PR 有相应的issue。
+3. 如果您的 PR 包含较大的更改，例如组件重构或新组件，请编写有关其设计和使用的详细文档(在对应的issue中)。
+4. 注意单个 PR 不能太大。如果需要进行大量更改，最好将更改分成几个单独的 PR。
+5. 在合并PR之前，尽量的将最终的提交信息清晰简洁, 将多次修改的提交尽可能的合并为一次提交。
+6. 创建 PR 后，将为PR分配一个或多个reviewers。
+
+
+<font color=red><b>如果您的 PR 包含较大的更改，例如组件重构或新组件，请编写有关其设计和使用的详细文档。</b></font>
+
+
+# 代码审查指南
+
+Commiter将轮流review代码，以确保在合并前至少有一名Commiter
+
+一些原则：
+
+- 可读性——重要的代码应该有详细的文档。API 应该有 Javadoc。代码风格应与现有风格保持一致。
+- 优雅：新的函数、类或组件应该设计得很好。
+- 可测试性——单元测试用例应该覆盖 80% 的新代码。
+- 可维护性 - 遵守我们的编码规范。
+
+
+# 开发者
+
+## 成为Contributor
+
+只要成功提交并合并PR , 则为Contributor
+
+贡献者名单请看：[贡献者名单](./docs/contributer_guide/开发者名单.md)
+
+## 尝试成为Commiter
+
+一般来说, 贡献8个重要的补丁并至少让三个不同的人来Review他们(您需要3个Commiter的支持)。
+然后请人给你提名, 您需要展示您的
+
+1. 至少8个重要的PR和项目的相关问题
+2. 与团队合作的能力
+3. 了解项目的代码库和编码风格
+4. 编写好代码的能力
+
+当前的Commiter可以通过在KnowStreaming中的Issue标签 `nomination`(提名)来提名您
+
+1. 你的名字和姓氏
+2. 指向您的Git个人资料的链接
+3. 解释为什么你应该成为Commiter
+4. 详细说明提名人与您合作的3个PR以及相关问题,这些问题可以证明您的能力。
+
+另外2个Commiter需要支持您的**提名**，如果5个工作日内没有人反对，您就是提交者,如果有人反对或者想要更多的信息，Commiter会讨论并通常达成共识(5个工作日内) 。
+
+
+# 开源奖励计划
+
+
+我们非常欢迎开发者们为KnowStreaming开源项目贡献一份力量，相应也将给予贡献者激励以表认可与感谢。
+
+
+## 参与贡献
+
+1. 积极参与 Issue 的讨论，如答疑解惑、提供想法或报告无法解决的错误（Issue）
+2. 撰写和改进项目的文档（Wiki）
+3. 提交补丁优化代码（Coding）
+
+
+## 你将获得
+
+1. 加入KnowStreaming开源项目贡献者名单并展示
+2. KnowStreaming开源贡献者证书(纸质&电子版)
+3. KnowStreaming贡献者精美大礼包(KnowStreamin/滴滴  周边)
+
+
+## 相关规则
+
+- Contributer和Commiter都会有对应的证书和对应的礼包
+- 每季度有KnowStreaming项目团队评选出杰出贡献者,颁发相应证书。
+- 年末进行年度评选
+
+贡献者名单请看：[贡献者名单](./docs/contributer_guide/开发者名单.md)

README.md

@@ -45,7 +45,14 @@
 
 ## `Know Streaming` 简介
 
-`Know Streaming`是一套云原生的Kafka管控平台，脱胎于众多互联网内部多年的Kafka运营实践经验，专注于Kafka运维管控、监控告警、资源治理、多活容灾等核心场景。在用户体验、监控、运维管控上进行了平台化、可视化、智能化的建设，提供一系列特色的功能，极大地方便了用户和运维人员的日常使用，让普通运维人员都能成为Kafka专家。整体具有以下特点：
+`Know Streaming`是一套云原生的Kafka管控平台，脱胎于众多互联网内部多年的Kafka运营实践经验，专注于Kafka运维管控、监控告警、资源治理、多活容灾等核心场景。在用户体验、监控、运维管控上进行了平台化、可视化、智能化的建设，提供一系列特色的功能，极大地方便了用户和运维人员的日常使用，让普通运维人员都能成为Kafka专家。
+
+我们现在正在收集 Know Streaming 用户信息，以帮助我们进一步改进  Know Streaming。
+请在 [issue#663](https://github.com/didi/KnowStreaming/issues/663) 上提供您的使用信息来支持我们：[谁在使用 Know Streaming](https://github.com/didi/KnowStreaming/issues/663) 
+
+
+
+整体具有以下特点：
 
 - 👀  **零侵入、全覆盖**
   - 无需侵入改造 `Apache Kafka` ，一键便能纳管 `0.10.x` ~ `3.x.x` 众多版本的Kafka，包括 `ZK` 或 `Raft` 运行模式的版本，同时在兼容架构上具备良好的扩展性，帮助您提升集群管理水平；
@@ -99,9 +106,13 @@
 
 ## 成为社区贡献者
 
-点击 [这里](CONTRIBUTING.md)，了解如何成为 Know Streaming 的贡献者
+1. [贡献源码](https://doc.knowstreaming.com/product/10-contribution) 了解如何成为 Know Streaming 的贡献者
+2. [具体贡献流程](https://doc.knowstreaming.com/product/10-contribution#102-贡献流程)
+3. [开源激励计划](https://doc.knowstreaming.com/product/10-contribution#105-开源激励计划)
+4. [贡献者名单](https://doc.knowstreaming.com/product/10-contribution#106-贡献者名单)
 
 
+获取KnowStreaming开源社区证书。
 
 ## 加入技术交流群
 
@@ -132,7 +143,14 @@ PS: 提问请尽量把问题一次性描述清楚，并告知环境信息情况
 
 **`2、微信群`**
 
-微信加群：添加`mike_zhangliang`、`PenceXie`的微信号备注KnowStreaming加群。
+微信加群：添加`mike_zhangliang`、`PenceXie` 、`szzdzhp001`的微信号备注KnowStreaming加群。
+<br/>
+
+加群之前有劳点一下 star，一个小小的 star 是对KnowStreaming作者们努力建设社区的动力。
+
+感谢感谢！！！
+
+<img width="116" alt="wx" src="https://user-images.githubusercontent.com/71620349/192257217-c4ebc16c-3ad9-485d-a914-5911d3a4f46b.png">
 
 ## Star History
 

Releases_Notes.md

@@ -1,4 +1,207 @@
 
+## v3.3.0
+
+**问题修复**
+- 修复 Connect 的 JMX-Port 配置未生效问题;
+- 修复 不存在 Connector 时，OverView 页面的数据一直处于加载中的问题;
+- 修复 Group 分区信息，分页时展示不全的问题；
+- 修复采集副本指标时，参数传递错误的问题；
+- 修复用户信息修改后，用户列表会抛出空指针异常的问题；
+- 修复 Topic 详情页面，查看消息时，选择分区不生效问题；
+- 修复对 ZK 客户端进行配置后不生效的问题；
+- 修复 connect 模块，指标中缺少健康巡检项通过数的问题；
+- 修复 connect 模块，指标获取方法存在映射错误的问题；
+- 修复 connect 模块，max 纬度指标获取错误的问题；
+- 修复 Topic 指标大盘 TopN 指标显示信息错误的问题；
+- 修复 Broker Similar Config 显示错误的问题；
+- 修复解析 ZK 四字命令时，数据类型设置错误导致空指针的问题；
+- 修复新增 Topic 时，清理策略选项版本控制错误的问题；
+- 修复新接入集群时 Controller-Host 信息不显示的问题；
+- 修复 Connector 和 MM2 列表搜索不生效的问题；
+- 修复 Zookeeper 页面，Leader 显示存在异常的问题；
+- 修复前端打包失败的问题；
+
+
+**产品优化**
+- ZK Overview 页面补充默认展示的指标；
+- 统一初始化 ES 索引模版的脚本为 init_es_template.sh，同时新增缺失的 connect 索引模版初始化脚本，去除多余的 replica 和 zookeper 索引模版初始化脚本；
+- 指标大盘页面，优化指标筛选操作后，无指标数据的指标卡片由不显示改为显示，并增加无数据的兜底；
+- 删除从 ES 读写 replica 指标的相关代码；
+- 优化 Topic 健康巡检的日志，明确错误的原因；
+- 优化无 ZK 模块时，巡检详情忽略对 ZK 的展示；
+- 优化本地缓存大小为可配置；
+- Task 模块中的返回中，补充任务的分组信息；
+- FAQ 补充 Ldap 的配置说明；
+- FAQ 补充接入 Kerberos 认证的 Kafka 集群的配置说明；
+- ks_km_kafka_change_record 表增加时间纬度的索引，优化查询性能；
+- 优化 ZK 健康巡检的日志，便于问题的排查；
+
+**功能新增**
+- 新增基于滴滴 Kafka 的 Topic 复制功能（需使用滴滴 Kafka 才可具备该能力）；
+- Topic 指标大盘，新增 Topic 复制相关的指标；
+- 新增基于 TestContainers 的单测；
+
+
+**Kafka MM2 Beta版 (v3.3.0版本新增发布)**
+- MM2 任务的增删改查；
+- MM2 任务的指标大盘；
+- MM2 任务的健康状态；
+
+---
+
+
+## v3.2.0
+
+**问题修复**
+- 修复健康巡检结果更新至 DB 时，出现死锁问题；
+- 修复 KafkaJMXClient 类中，logger错误的问题；
+- 后端修复 Topic 过期策略在 0.10.1.0 版本能多选的问题，实际应该只能二选一；
+- 修复接入集群时，不填写集群配置会报错的问题；
+- 升级 spring-context 至 5.3.19 版本，修复安全漏洞；
+- 修复 Broker & Topic 修改配置时，多版本兼容配置的版本信息错误的问题；
+- 修复 Topic 列表的健康分为健康状态；
+- 修复 Broker LogSize 指标存储名称错误导致查询不到的问题；
+- 修复 Prometheus 中，缺少 Group 部分指标的问题；
+- 修复因缺少健康状态指标导致集群数错误的问题；
+- 修复后台任务记录操作日志时，因缺少操作用户信息导致出现异常的问题；
+- 修复 Replica 指标查询时，DSL 错误的问题；
+- 关闭 errorLogger，修复错误日志重复输出的问题；
+- 修复系统管理更新用户信息失败的问题；
+- 修复因原AR信息丢失，导致迁移任务一直处于执行中的错误；
+- 修复集群 Topic 列表实时数据查询时，出现失败的问题；
+- 修复集群 Topic 列表，页面白屏问题；
+- 修复副本变更时，因AR数据异常，导致数组访问越界的问题；
+
+
+**产品优化**
+- 优化健康巡检为按照资源维度多线程并发处理；
+- 统一日志输出格式，并优化部分输出的日志；
+- 优化 ZK 四字命令结果解析过程中，容易引起误解的 WARN 日志；
+- 优化 Zookeeper 详情中，目录结构的搜索文案；
+- 优化线程池的名称，方便第三方系统进行相关问题的分析；
+- 去除 ESClient 的并发访问控制，降低 ESClient 创建数及提升利用率；
+- 优化 Topic Messages 抽屉文案；
+- 优化 ZK 健康巡检失败时的错误日志信息；
+- 提高 Offset 信息获取的超时时间，降低并发过高时出现请求超时的概率；
+- 优化 Topic & Partition 元信息的更新策略，降低对 DB 连接的占用；
+- 优化 Sonar 代码扫码问题；
+- 优化分区 Offset 指标的采集；
+- 优化前端图表相关组件逻辑;
+- 优化产品主题色;
+- Consumer 列表刷新按钮新增 hover 提示；
+- 优化配置 Topic 的消息大小时的测试弹框体验；
+- 优化 Overview 页面 TopN 查询的流程；
+
+
+**功能新增**
+- 新增页面无数据排查文档；
+- 增加 ES 索引删除的功能；
+- 支持拆分API服务和Job服务部署；
+
+
+**Kafka Connect Beta版 (v3.2.0版本新增发布)**
+- Connect 集群的纳管；
+- Connector 的增删改查；
+- Connect 集群 & Connector 的指标大盘；
+
+
+---
+
+
+## v3.1.0
+
+**Bug修复**
+- 修复重置 Group Offset 的提示信息中，缺少Dead状态也可进行重置的描述；
+- 修复新建 Topic 后，立即查看 Topic Messages 信息时，会提示 Topic 不存在的问题；
+- 修复副本变更时，优先副本选举未被正常处罚执行的问题；
+- 修复 git 目录不存在时，打包不能正常进行的问题；
+- 修复 KRaft 模式的 Kafka 集群，JMX PORT 显示 -1 的问题；
+
+
+**体验优化**
+- 优化Cluster、Broker、Topic、Group的健康分为健康状态；
+- 去除健康巡检配置中的权重信息；
+- 错误提示页面展示优化；
+- 前端打包编译依赖默认使用 taobao 镜像；
+- 重新设计优化导航栏的 icon ；
+
+
+**新增**
+- 个人头像下拉信息中，新增产品版本信息；
+- 多集群列表页面，新增集群健康状态分布信息；
+
+
+**Kafka ZK 部分 (v3.1.0版本正式发布)**
+- 新增 ZK 集群的指标大盘信息；
+- 新增 ZK 集群的服务状态概览信息；
+- 新增 ZK 集群的服务节点列表信息；
+- 新增 Kafka 在 ZK 的存储数据查看功能；
+- 新增 ZK 的健康巡检及健康状态计算；
+
+
+
+---
+
+
+## v3.0.1
+
+**Bug修复**
+- 修复重置 Group Offset 时，提示信息中缺少 Dead 状态也可进行重置的信息；
+- 修复 Ldap 某个属性不存在时，会直接抛出空指针导致登陆失败的问题；
+- 修复集群 Topic 列表页，健康分详情信息中，检查时间展示错误的问题；
+- 修复更新健康检查结果时，出现死锁的问题；
+- 修复 Replica 索引模版错误的问题；
+- 修复 FAQ 文档中的错误链接；
+- 修复 Broker 的 TopN 指标不存在时，页面数据不展示的问题；
+- 修复 Group 详情页，图表时间范围选择不生效的问题；
+
+
+**体验优化**
+- 集群 Group 列表按照 Group 维度进行展示；
+- 优化避免因 ES 中该指标不存在，导致日志中出现大量空指针的问题；
+- 优化全局 Message & Notification 展示效果；
+- 优化 Topic 扩分区名称 & 描述展示；
+
+
+**新增**
+- Broker 列表页面，新增 JMX 是否成功连接的信息；
+
+
+**ZK 部分(未完全发布)**
+- 后端补充 Kafka ZK 指标采集，Kafka ZK 信息获取相关功能；
+- 增加本地缓存，避免同一采集周期内 ZK 指标重复采集；
+- 增加 ZK 节点采集失败跳过策略，避免不断对存在问题的节点不断尝试；
+- 修复 zkAvgLatency 指标转 Long 时抛出异常问题；
+- 修复 ks_km_zookeeper 表中，role 字段类型错误问题；
+
+---
+
+## v3.0.0
+
+**Bug修复**
+- 修复 Group 指标防重复采集不生效问题
+- 修复自动创建 ES 索引模版失败问题
+- 修复 Group+Topic 列表中存在已删除Topic的问题
+- 修复使用 MySQL-8 ，因兼容问题， start_time 信息为 NULL 时，会导致创建任务失败的问题
+- 修复 Group 信息表更新时，出现死锁的问题
+- 修复图表补点逻辑与图表时间范围不适配的问题
+
+
+**体验优化**
+- 按照资源类别，拆分健康巡检任务
+- 优化 Group 详情页的指标为实时获取
+- 图表拖拽排序支持用户级存储
+- 多集群列表 ZK 信息展示兼容无 ZK 情况
+- Topic 详情消息预览支持复制功能
+- 部分内容大数字支持千位分割符展示
+
+
+**新增**
+- 集群信息中，新增 Zookeeper 客户端配置字段
+- 集群信息中，新增 Kafka 集群运行模式字段
+- 新增 docker-compose 的部署方式
+
+---
 
 ## v3.0.0-beta.3
 

bin/init_es_template.sh

@@ -13,7 +13,7 @@ curl -s --connect-timeout 10 -o /dev/null -X POST -H 'cache-control: no-cache' -
     ],
     "settings" : {
       "index" : {
-        "number_of_shards" : "10"
+        "number_of_shards" : "2"
       }
     },
     "mappings" : {
@@ -115,7 +115,7 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
     ],
     "settings" : {
       "index" : {
-        "number_of_shards" : "10"
+        "number_of_shards" : "2"
       }
     },
     "mappings" : {
@@ -302,7 +302,7 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
     ],
     "settings" : {
       "index" : {
-        "number_of_shards" : "10"
+        "number_of_shards" : "6"
       }
     },
     "mappings" : {
@@ -377,7 +377,7 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
     ],
     "settings" : {
       "index" : {
-        "number_of_shards" : "10"
+        "number_of_shards" : "6"
       }
     },
     "mappings" : {
@@ -436,95 +436,6 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
     "aliases" : { }
   }' 
 
-curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: application/json' http://${esaddr}:${port}/_template/ks_kafka_replication_metric -d '{
-    "order" : 10,
-    "index_patterns" : [
-      "ks_kafka_partition_metric*"
-    ],
-    "settings" : {
-      "index" : {
-        "number_of_shards" : "10"
-      }
-    },
-    "mappings" : {
-      "properties" : {
-        "brokerId" : {
-          "type" : "long"
-        },
-        "partitionId" : {
-          "type" : "long"
-        },
-        "routingValue" : {
-          "type" : "text",
-          "fields" : {
-            "keyword" : {
-              "ignore_above" : 256,
-              "type" : "keyword"
-            }
-          }
-        },
-        "clusterPhyId" : {
-          "type" : "long"
-        },
-        "topic" : {
-          "type" : "keyword"
-        },
-        "metrics" : {
-          "properties" : {
-            "LogStartOffset" : {
-              "type" : "float"
-            },
-            "Messages" : {
-              "type" : "float"
-            },
-            "LogEndOffset" : {
-              "type" : "float"
-            }
-          }
-        },
-        "key" : {
-          "type" : "text",
-          "fields" : {
-            "keyword" : {
-              "ignore_above" : 256,
-              "type" : "keyword"
-            }
-          }
-        },
-        "timestamp" : {
-          "format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
-          "index" : true,
-          "type" : "date",
-          "doc_values" : true
-        }
-      }
-    },
-    "aliases" : { }
-  }[root@10-255-0-23 template]# cat ks_kafka_replication_metric
-PUT _template/ks_kafka_replication_metric
-{
-    "order" : 10,
-    "index_patterns" : [
-      "ks_kafka_replication_metric*"
-    ],
-    "settings" : {
-      "index" : {
-        "number_of_shards" : "10"
-      }
-    },
-    "mappings" : {
-      "properties" : {
-        "timestamp" : {
-          "format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
-          "index" : true,
-          "type" : "date",
-          "doc_values" : true
-        }
-      }
-    },
-    "aliases" : { }
-  }' 
-
 curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: application/json' http://${esaddr}:${port}/_template/ks_kafka_topic_metric -d '{
     "order" : 10,
     "index_patterns" : [
@@ -532,7 +443,7 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
     ],
     "settings" : {
       "index" : {
-        "number_of_shards" : "10"
+        "number_of_shards" : "6"
       }
     },
     "mappings" : {
@@ -640,7 +551,474 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
       }
     },
     "aliases" : { }
-  }' 
+  }'
+
+curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: application/json' http://${SERVER_ES_ADDRESS}/_template/ks_kafka_zookeeper_metric -d '{
+    "order" : 10,
+    "index_patterns" : [
+      "ks_kafka_zookeeper_metric*"
+    ],
+    "settings" : {
+      "index" : {
+        "number_of_shards" : "2"
+      }
+    },
+    "mappings" : {
+      "properties" : {
+        "routingValue" : {
+          "type" : "text",
+          "fields" : {
+            "keyword" : {
+              "ignore_above" : 256,
+              "type" : "keyword"
+            }
+          }
+        },
+        "clusterPhyId" : {
+          "type" : "long"
+        },
+        "metrics" : {
+          "properties" : {
+            "AvgRequestLatency" : {
+              "type" : "double"
+            },
+            "MinRequestLatency" : {
+              "type" : "double"
+            },
+            "MaxRequestLatency" : {
+              "type" : "double"
+            },
+            "OutstandingRequests" : {
+              "type" : "double"
+            },
+            "NodeCount" : {
+              "type" : "double"
+            },
+            "WatchCount" : {
+              "type" : "double"
+            },
+            "NumAliveConnections" : {
+              "type" : "double"
+            },
+            "PacketsReceived" : {
+              "type" : "double"
+            },
+            "PacketsSent" : {
+              "type" : "double"
+            },
+            "EphemeralsCount" : {
+              "type" : "double"
+            },
+            "ApproximateDataSize" : {
+              "type" : "double"
+            },
+            "OpenFileDescriptorCount" : {
+              "type" : "double"
+            },
+            "MaxFileDescriptorCount" : {
+              "type" : "double"
+            }
+          }
+        },
+        "key" : {
+          "type" : "text",
+          "fields" : {
+            "keyword" : {
+              "ignore_above" : 256,
+              "type" : "keyword"
+            }
+          }
+        },
+        "timestamp" : {
+          "format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
+          "type" : "date"
+        }
+      }
+    },
+    "aliases" : { }
+  }'
+
+curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: application/json' http://${SERVER_ES_ADDRESS}/_template/ks_kafka_connect_cluster_metric -d '{
+    "order" : 10,
+    "index_patterns" : [
+      "ks_kafka_connect_cluster_metric*"
+    ],
+    "settings" : {
+      "index" : {
+        "number_of_shards" : "2"
+      }
+    },
+    "mappings" : {
+      "properties" : {
+        "connectClusterId" : {
+          "type" : "long"
+        },
+        "routingValue" : {
+          "type" : "text",
+          "fields" : {
+            "keyword" : {
+              "ignore_above" : 256,
+              "type" : "keyword"
+            }
+          }
+        },
+        "clusterPhyId" : {
+          "type" : "long"
+        },
+        "metrics" : {
+          "properties" : {
+            "ConnectorCount" : {
+              "type" : "float"
+            },
+            "TaskCount" : {
+              "type" : "float"
+            },
+            "ConnectorStartupAttemptsTotal" : {
+              "type" : "float"
+            },
+            "ConnectorStartupFailurePercentage" : {
+              "type" : "float"
+            },
+            "ConnectorStartupFailureTotal" : {
+              "type" : "float"
+            },
+            "ConnectorStartupSuccessPercentage" : {
+              "type" : "float"
+            },
+            "ConnectorStartupSuccessTotal" : {
+              "type" : "float"
+            },
+            "TaskStartupAttemptsTotal" : {
+              "type" : "float"
+            },
+            "TaskStartupFailurePercentage" : {
+              "type" : "float"
+            },
+            "TaskStartupFailureTotal" : {
+              "type" : "float"
+            },
+            "TaskStartupSuccessPercentage" : {
+              "type" : "float"
+            },
+            "TaskStartupSuccessTotal" : {
+              "type" : "float"
+            }
+          }
+        },
+        "key" : {
+          "type" : "text",
+          "fields" : {
+            "keyword" : {
+              "ignore_above" : 256,
+              "type" : "keyword"
+            }
+          }
+        },
+        "timestamp" : {
+          "format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
+          "index" : true,
+          "type" : "date",
+          "doc_values" : true
+        }
+      }
+    },
+    "aliases" : { }
+  }'
+
+curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: application/json' http://${SERVER_ES_ADDRESS}/_template/ks_kafka_connect_connector_metric -d '{
+    "order" : 10,
+    "index_patterns" : [
+      "ks_kafka_connect_connector_metric*"
+    ],
+    "settings" : {
+      "index" : {
+        "number_of_shards" : "2"
+      }
+    },
+    "mappings" : {
+      "properties" : {
+        "connectClusterId" : {
+          "type" : "long"
+        },
+        "routingValue" : {
+          "type" : "text",
+          "fields" : {
+            "keyword" : {
+              "ignore_above" : 256,
+              "type" : "keyword"
+            }
+          }
+        },
+        "connectorName" : {
+          "type" : "keyword"
+        },
+        "connectorNameAndClusterId" : {
+          "type" : "keyword"
+        },
+        "clusterPhyId" : {
+          "type" : "long"
+        },
+        "metrics" : {
+          "properties" : {
+            "HealthState" : {
+              "type" : "float"
+            },
+            "ConnectorTotalTaskCount" : {
+              "type" : "float"
+            },
+            "HealthCheckPassed" : {
+              "type" : "float"
+            },
+            "HealthCheckTotal" : {
+              "type" : "float"
+            },
+            "ConnectorRunningTaskCount" : {
+              "type" : "float"
+            },
+            "ConnectorPausedTaskCount" : {
+              "type" : "float"
+            },
+            "ConnectorFailedTaskCount" : {
+              "type" : "float"
+            },
+            "ConnectorUnassignedTaskCount" : {
+              "type" : "float"
+            },
+            "BatchSizeAvg" : {
+              "type" : "float"
+            },
+            "BatchSizeMax" : {
+              "type" : "float"
+            },
+            "OffsetCommitAvgTimeMs" : {
+              "type" : "float"
+            },
+            "OffsetCommitMaxTimeMs" : {
+              "type" : "float"
+            },
+             "OffsetCommitFailurePercentage" : {
+               "type" : "float"
+             },
+             "OffsetCommitSuccessPercentage" : {
+               "type" : "float"
+             },
+             "PollBatchAvgTimeMs" : {
+               "type" : "float"
+             },
+             "PollBatchMaxTimeMs" : {
+               "type" : "float"
+             },
+             "SourceRecordActiveCount" : {
+               "type" : "float"
+             },
+             "SourceRecordActiveCountAvg" : {
+               "type" : "float"
+             },
+             "SourceRecordActiveCountMax" : {
+               "type" : "float"
+             },
+             "SourceRecordPollRate" : {
+               "type" : "float"
+             },
+             "SourceRecordPollTotal" : {
+               "type" : "float"
+             },
+             "SourceRecordWriteRate" : {
+               "type" : "float"
+             },
+             "SourceRecordWriteTotal" : {
+               "type" : "float"
+             },
+           "OffsetCommitCompletionRate" : {
+             "type" : "float"
+           },
+           "OffsetCommitCompletionTotal" : {
+             "type" : "float"
+           },
+           "OffsetCommitSkipRate" : {
+             "type" : "float"
+           },
+           "OffsetCommitSkipTotal" : {
+             "type" : "float"
+           },
+           "PartitionCount" : {
+             "type" : "float"
+           },
+           "PutBatchAvgTimeMs" : {
+             "type" : "float"
+           },
+           "PutBatchMaxTimeMs" : {
+             "type" : "float"
+           },
+           "SinkRecordActiveCount" : {
+             "type" : "float"
+           },
+           "SinkRecordActiveCountAvg" : {
+             "type" : "float"
+           },
+           "SinkRecordActiveCountMax" : {
+             "type" : "float"
+           },
+           "SinkRecordLagMax" : {
+             "type" : "float"
+           },
+           "SinkRecordReadRate" : {
+             "type" : "float"
+           },
+           "SinkRecordReadTotal" : {
+             "type" : "float"
+           },
+           "SinkRecordSendRate" : {
+             "type" : "float"
+           },
+           "SinkRecordSendTotal" : {
+             "type" : "float"
+           },
+           "DeadletterqueueProduceFailures" : {
+             "type" : "float"
+           },
+           "DeadletterqueueProduceRequests" : {
+             "type" : "float"
+           },
+           "LastErrorTimestamp" : {
+             "type" : "float"
+           },
+           "TotalErrorsLogged" : {
+             "type" : "float"
+           },
+           "TotalRecordErrors" : {
+             "type" : "float"
+           },
+           "TotalRecordFailures" : {
+             "type" : "float"
+           },
+           "TotalRecordsSkipped" : {
+             "type" : "float"
+           },
+           "TotalRetries" : {
+             "type" : "float"
+           }
+          }
+        },
+        "key" : {
+          "type" : "text",
+          "fields" : {
+            "keyword" : {
+              "ignore_above" : 256,
+              "type" : "keyword"
+            }
+          }
+        },
+        "timestamp" : {
+          "format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
+          "index" : true,
+          "type" : "date",
+          "doc_values" : true
+        }
+      }
+    },
+    "aliases" : { }
+  }'
+
+curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: application/json' http://${SERVER_ES_ADDRESS}/_template/ks_kafka_connect_mirror_maker_metric -d '{
+    "order" : 10,
+    "index_patterns" : [
+      "ks_kafka_connect_mirror_maker_metric*"
+    ],
+    "settings" : {
+      "index" : {
+        "number_of_shards" : "2"
+      }
+    },
+    "mappings" : {
+      "properties" : {
+        "connectClusterId" : {
+          "type" : "long"
+        },
+        "routingValue" : {
+          "type" : "text",
+          "fields" : {
+            "keyword" : {
+              "ignore_above" : 256,
+              "type" : "keyword"
+            }
+          }
+        },
+        "connectorName" : {
+          "type" : "keyword"
+        },
+        "connectorNameAndClusterId" : {
+          "type" : "keyword"
+        },
+        "clusterPhyId" : {
+          "type" : "long"
+        },
+        "metrics" : {
+          "properties" : {
+            "HealthState" : {
+              "type" : "float"
+            },
+            "HealthCheckTotal" : {
+              "type" : "float"
+            },
+            "ByteCount" : {
+              "type" : "float"
+            },
+            "ByteRate" : {
+              "type" : "float"
+            },
+            "RecordAgeMs" : {
+              "type" : "float"
+            },
+            "RecordAgeMsAvg" : {
+              "type" : "float"
+            },
+            "RecordAgeMsMax" : {
+              "type" : "float"
+            },
+            "RecordAgeMsMin" : {
+              "type" : "float"
+            },
+            "RecordCount" : {
+              "type" : "float"
+            },
+            "RecordRate" : {
+              "type" : "float"
+            },
+            "ReplicationLatencyMs" : {
+              "type" : "float"
+            },
+            "ReplicationLatencyMsAvg" : {
+              "type" : "float"
+            },
+            "ReplicationLatencyMsMax" : {
+               "type" : "float"
+            },
+            "ReplicationLatencyMsMin" : {
+               "type" : "float"
+            }
+          }
+        },
+        "key" : {
+          "type" : "text",
+          "fields" : {
+            "keyword" : {
+              "ignore_above" : 256,
+              "type" : "keyword"
+            }
+          }
+        },
+        "timestamp" : {
+          "format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
+          "index" : true,
+          "type" : "date",
+          "doc_values" : true
+        }
+      }
+    },
+    "aliases" : { }
+  }'
+
 
 for i in {0..6};
 do
@@ -649,7 +1027,10 @@ do
     curl -s -o /dev/null -X PUT http://${esaddr}:${port}/ks_kafka_cluster_metric${logdate} && \
     curl -s -o /dev/null -X PUT http://${esaddr}:${port}/ks_kafka_group_metric${logdate} && \
     curl -s -o /dev/null -X PUT http://${esaddr}:${port}/ks_kafka_partition_metric${logdate} && \
-    curl -s -o /dev/null -X PUT http://${esaddr}:${port}/ks_kafka_replication_metric${logdate} && \
+    curl -s -o /dev/null -X PUT http://${esaddr}:${port}/ks_kafka_zookeeper_metric${logdate} && \
+    curl -s -o /dev/null -X PUT http://${esaddr}:${port}/ks_kafka_connect_cluster_metric${logdate} && \
+    curl -s -o /dev/null -X PUT http://${esaddr}:${port}/ks_kafka_connect_connector_metric${logdate} && \
+    curl -s -o /dev/null -X PUT http://${esaddr}:${port}/ks_kafka_connect_mirror_maker_metric${logdate} && \
     curl -s -o /dev/null -X PUT http://${esaddr}:${port}/ks_kafka_topic_metric${logdate} || \
     exit 2
-done
+done

docs/contribute_guide/assets/分支管理.drawio

@@ -0,0 +1,111 @@
+<mxfile host="65bd71144e">
+    <diagram id="vxzhwhZdNVAY19FZ4dgb" name="Page-1">
+        <mxGraphModel dx="1194" dy="733" grid="0" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1169" pageHeight="827" math="0" shadow="0">
+            <root>
+                <mxCell id="0"/>
+                <mxCell id="1" parent="0"/>
+                <mxCell id="4" style="edgeStyle=none;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;startArrow=none;strokeWidth=2;strokeColor=#6666FF;" edge="1" parent="1" source="16">
+                    <mxGeometry relative="1" as="geometry">
+                        <mxPoint x="200" y="540" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="7" style="edgeStyle=none;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;exitPerimeter=0;strokeColor=#33FF33;strokeWidth=2;" edge="1" parent="1" source="2">
+                    <mxGeometry relative="1" as="geometry">
+                        <mxPoint x="360" y="240" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="5" style="edgeStyle=none;html=1;startArrow=none;strokeColor=#33FF33;strokeWidth=2;" edge="1" parent="1">
+                    <mxGeometry relative="1" as="geometry">
+                        <mxPoint x="200" y="400" as="targetPoint"/>
+                        <mxPoint x="360" y="360" as="sourcePoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="3" value="C3" style="verticalLabelPosition=middle;verticalAlign=middle;html=1;shape=mxgraph.flowchart.on-page_reference;labelPosition=center;align=center;strokeColor=#FF8000;strokeWidth=2;" vertex="1" parent="1">
+                    <mxGeometry x="340" y="280" width="40" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="18" style="edgeStyle=none;html=1;entryX=0.5;entryY=0;entryDx=0;entryDy=0;entryPerimeter=0;endArrow=none;endFill=0;strokeColor=#FF8000;strokeWidth=2;" edge="1" parent="1" source="8" target="3">
+                    <mxGeometry relative="1" as="geometry"/>
+                </mxCell>
+                <mxCell id="8" value="fix_928" style="rounded=1;whiteSpace=wrap;html=1;absoluteArcSize=1;arcSize=14;strokeWidth=0;" vertex="1" parent="1">
+                    <mxGeometry x="320" y="40" width="80" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="9" value="github_master" style="rounded=1;whiteSpace=wrap;html=1;absoluteArcSize=1;arcSize=14;strokeWidth=0;" vertex="1" parent="1">
+                    <mxGeometry x="160" y="40" width="80" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="10" value="" style="edgeStyle=none;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;endArrow=classic;startArrow=none;endFill=1;strokeWidth=2;strokeColor=#6666FF;" edge="1" parent="1" source="11" target="2">
+                    <mxGeometry relative="1" as="geometry">
+                        <mxPoint x="200" y="640" as="targetPoint"/>
+                        <mxPoint x="200" y="80" as="sourcePoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="2" value="C2" style="verticalLabelPosition=middle;verticalAlign=middle;html=1;shape=mxgraph.flowchart.on-page_reference;labelPosition=center;align=center;strokeColor=#6666FF;strokeWidth=2;" vertex="1" parent="1">
+                    <mxGeometry x="180" y="200" width="40" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="12" value="" style="edgeStyle=none;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;endArrow=classic;endFill=1;strokeWidth=2;strokeColor=#6666FF;" edge="1" parent="1" source="9" target="11">
+                    <mxGeometry relative="1" as="geometry">
+                        <mxPoint x="200" y="200" as="targetPoint"/>
+                        <mxPoint x="200" y="80" as="sourcePoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="11" value="C1" style="verticalLabelPosition=middle;verticalAlign=middle;html=1;shape=mxgraph.flowchart.on-page_reference;labelPosition=center;align=center;strokeColor=#6666FF;strokeWidth=2;" vertex="1" parent="1">
+                    <mxGeometry x="180" y="120" width="40" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="23" style="edgeStyle=none;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;exitPerimeter=0;endArrow=none;endFill=0;strokeColor=#FF8000;strokeWidth=2;" edge="1" parent="1" source="3">
+                    <mxGeometry relative="1" as="geometry">
+                        <mxPoint x="360" y="360" as="targetPoint"/>
+                        <mxPoint x="360" y="400" as="sourcePoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="17" value="" style="edgeStyle=none;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;startArrow=none;endArrow=none;strokeWidth=2;strokeColor=#6666FF;" edge="1" parent="1" source="2" target="16">
+                    <mxGeometry relative="1" as="geometry">
+                        <mxPoint x="200" y="640" as="targetPoint"/>
+                        <mxPoint x="200" y="240" as="sourcePoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="16" value="C4" style="verticalLabelPosition=middle;verticalAlign=middle;html=1;shape=mxgraph.flowchart.on-page_reference;labelPosition=center;align=center;strokeColor=#6666FF;strokeWidth=2;" vertex="1" parent="1">
+                    <mxGeometry x="180" y="440" width="40" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="22" value="Tag-v3.2.0" style="rounded=0;whiteSpace=wrap;html=1;absoluteArcSize=1;arcSize=14;strokeWidth=0;fillColor=none;strokeColor=none;" vertex="1" parent="1">
+                    <mxGeometry x="100" y="120" width="80" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="24" value="Tag-v3.2.1" style="rounded=0;whiteSpace=wrap;html=1;absoluteArcSize=1;arcSize=14;strokeWidth=0;fillColor=none;strokeColor=none;" vertex="1" parent="1">
+                    <mxGeometry x="100" y="440" width="80" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="27" value="切换到主分支：git checkout github_master" style="rounded=0;whiteSpace=wrap;html=1;absoluteArcSize=1;arcSize=14;strokeWidth=0;labelPosition=center;verticalLabelPosition=middle;align=center;verticalAlign=middle;" vertex="1" parent="1">
+                    <mxGeometry x="520" y="90" width="240" height="30" as="geometry"/>
+                </mxCell>
+                <mxCell id="34" style="edgeStyle=none;html=1;exitX=0;exitY=0;exitDx=0;exitDy=0;entryX=0.855;entryY=0.145;entryDx=0;entryDy=0;entryPerimeter=0;dashed=1;dashPattern=8 8;fontSize=18;endArrow=none;endFill=0;" edge="1" parent="1" source="28" target="2">
+                    <mxGeometry relative="1" as="geometry"/>
+                </mxCell>
+                <mxCell id="28" value="主分支拉最新代码：git pull" style="rounded=0;whiteSpace=wrap;html=1;absoluteArcSize=1;arcSize=14;strokeWidth=0;labelPosition=center;verticalLabelPosition=middle;align=center;verticalAlign=middle;" vertex="1" parent="1">
+                    <mxGeometry x="520" y="120" width="160" height="30" as="geometry"/>
+                </mxCell>
+                <mxCell id="35" style="edgeStyle=none;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;dashed=1;dashPattern=8 8;fontSize=18;endArrow=none;endFill=0;" edge="1" parent="1" source="29">
+                    <mxGeometry relative="1" as="geometry">
+                        <mxPoint x="270" y="225" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="29" value="基于主分支拉新分支：git checkout -b fix_928" style="rounded=0;whiteSpace=wrap;html=1;absoluteArcSize=1;arcSize=14;strokeWidth=0;labelPosition=center;verticalLabelPosition=middle;align=center;verticalAlign=middle;" vertex="1" parent="1">
+                    <mxGeometry x="520" y="210" width="250" height="30" as="geometry"/>
+                </mxCell>
+                <mxCell id="37" style="edgeStyle=none;html=1;exitX=0;exitY=1;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;entryPerimeter=0;dashed=1;dashPattern=8 8;fontSize=18;endArrow=none;endFill=0;" edge="1" parent="1" source="30" target="3">
+                    <mxGeometry relative="1" as="geometry"/>
+                </mxCell>
+                <mxCell id="30" value="提交代码：git commit -m &quot;[Optimize]优化xxx问题(#928)&quot;" style="rounded=0;whiteSpace=wrap;html=1;absoluteArcSize=1;arcSize=14;strokeWidth=0;labelPosition=center;verticalLabelPosition=middle;align=center;verticalAlign=middle;" vertex="1" parent="1">
+                    <mxGeometry x="520" y="270" width="320" height="30" as="geometry"/>
+                </mxCell>
+                <mxCell id="31" value="提交到自己远端仓库：git push --set-upstream origin fix_928" style="rounded=0;whiteSpace=wrap;html=1;absoluteArcSize=1;arcSize=14;strokeWidth=0;labelPosition=center;verticalLabelPosition=middle;align=center;verticalAlign=middle;" vertex="1" parent="1">
+                    <mxGeometry x="520" y="300" width="334" height="30" as="geometry"/>
+                </mxCell>
+                <mxCell id="38" style="edgeStyle=none;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;dashed=1;dashPattern=8 8;fontSize=18;endArrow=none;endFill=0;" edge="1" parent="1" source="32">
+                    <mxGeometry relative="1" as="geometry">
+                        <mxPoint x="280" y="380" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="32" value="GitHub页面发起Pull Request请求，管理员合入主仓库" style="rounded=0;whiteSpace=wrap;html=1;absoluteArcSize=1;arcSize=14;strokeWidth=0;labelPosition=center;verticalLabelPosition=middle;align=center;verticalAlign=middle;" vertex="1" parent="1">
+                    <mxGeometry x="520" y="360" width="300" height="30" as="geometry"/>
+                </mxCell>
+            </root>
+        </mxGraphModel>
+    </diagram>
+</mxfile>

docs/contribute_guide/代码规范.md

@@ -0,0 +1 @@
+TODO.

docs/contribute_guide/贡献名单.md

@@ -0,0 +1,100 @@
+# 贡献名单
+
+- [贡献名单](#贡献名单)
+  - [1、贡献者角色](#1贡献者角色)
+    - [1.1、Maintainer](#11maintainer)
+    - [1.2、Committer](#12committer)
+    - [1.3、Contributor](#13contributor)
+  - [2、贡献者名单](#2贡献者名单)
+
+
+## 1、贡献者角色
+
+KnowStreaming 开发者包含 Maintainer、Committer、Contributor 三种角色，每种角色的标准定义如下。
+
+### 1.1、Maintainer
+
+Maintainer 是对 KnowStreaming 项目的演进和发展做出显著贡献的个人。具体包含以下的标准：
+
+- 完成多个关键模块或者工程的设计与开发，是项目的核心开发人员；
+- 持续的投入和激情，能够积极参与社区、官网、issue、PR 等项目相关事项的维护；
+- 在社区中具有有目共睹的影响力，能够代表 KnowStreaming 参加重要的社区会议和活动；
+- 具有培养 Committer 和 Contributor 的意识和能力；
+
+### 1.2、Committer
+
+Committer 是具有 KnowStreaming 仓库写权限的个人，包含以下的标准：
+
+- 能够在长时间内做持续贡献 issue、PR 的个人；
+- 参与 issue 列表的维护及重要 feature 的讨论；
+- 参与 code review；
+
+### 1.3、Contributor
+
+Contributor 是对 KnowStreaming 项目有贡献的个人，标准为：
+
+- 提交过 PR 并被合并；
+
+---
+
+## 2、贡献者名单
+
+开源贡献者名单(不定期更新)
+
+在名单内,但是没有收到贡献者礼品的同学,可以联系：szzdzhp001
+
+| 姓名                | Github                                                     | 角色        | 公司     |
+| ------------------- | ---------------------------------------------------------- | ----------- | -------- |
+| 张亮                | [@zhangliangboy](https://github.com/zhangliangboy)         | Maintainer  | 滴滴出行 |
+| 谢鹏                | [@PenceXie](https://github.com/PenceXie)                   | Maintainer  | 滴滴出行 |
+| 赵情融              | [@zqrferrari](https://github.com/zqrferrari)               | Maintainer  | 滴滴出行 |
+| 石臻臻              | [@shirenchuang](https://github.com/shirenchuang)           | Maintainer  | 滴滴出行 |
+| 曾巧                | [@ZQKC](https://github.com/ZQKC)                           | Maintainer  | 滴滴出行 |
+| 孙超                | [@lucasun](https://github.com/lucasun)                     | Maintainer  | 滴滴出行 |
+| 洪华驰              | [@brodiehong](https://github.com/brodiehong)               | Maintainer  | 滴滴出行 |
+| 许喆                | [@potaaaaaato](https://github.com/potaaaaaato)             | Committer   | 滴滴出行 |
+| 郭宇航              | [@GraceWalk](https://github.com/GraceWalk)                 | Committer   | 滴滴出行 |
+| 李伟                | [@velee](https://github.com/velee)                         | Committer   | 滴滴出行 |
+| 张占昌              | [@zzccctv](https://github.com/zzccctv)                     | Committer   | 滴滴出行 |
+| 王东方              | [@wangdongfang-aden](https://github.com/wangdongfang-aden) | Committer   | 滴滴出行 |
+| 王耀波              | [@WYAOBO](https://github.com/WYAOBO)                       | Committer   | 滴滴出行 |
+| 赵寅锐              | [@ZHAOYINRUI](https://github.com/ZHAOYINRUI)               | Maintainer  | 字节跳动 |
+| haoqi123            | [@haoqi123](https://github.com/haoqi123)                   | Contributor | 前程无忧 |
+| chaixiaoxue         | [@chaixiaoxue](https://github.com/chaixiaoxue)             | Contributor | SYNNEX   |
+| 陆晗                | [@luhea](https://github.com/luhea)                         | Contributor | 竞技世界 |
+| Mengqi777           | [@Mengqi777](https://github.com/Mengqi777)                 | Contributor | 腾讯     |
+| ruanliang-hualun    | [@ruanliang-hualun](https://github.com/ruanliang-hualun)   | Contributor | 网易     |
+| 17hao               | [@17hao](https://github.com/17hao)                         | Contributor |          |
+| Huyueeer            | [@Huyueeer](https://github.com/Huyueeer)                   | Contributor | INVENTEC |
+| lomodays207         | [@lomodays207](https://github.com/lomodays207)             | Contributor | 建信金科 |
+| Super .Wein（星痕） | [@superspeedone](https://github.com/superspeedone)         | Contributor | 韵达     |
+| Hongten             | [@Hongten](https://github.com/Hongten)                     | Contributor | Shopee   |
+| 徐正熙              | [@hyper-xx)](https://github.com/hyper-xx)                  | Contributor | 滴滴出行 |
+| RichardZhengkay     | [@RichardZhengkay](https://github.com/RichardZhengkay)     | Contributor | 趣街     |
+| 罐子里的茶          | [@gzldc](https://github.com/gzldc)                         | Contributor | 道富     |
+| 陈忠玉              | [@paula](https://github.com/chenzhongyu11)                 | Contributor | 平安产险 |
+| 杨光                | [@yaangvipguang](https://github.com/yangvipguang)          | Contributor |
+| 王亚聪              | [@wangyacongi](https://github.com/wangyacongi)             | Contributor |
+| Yang Jing           | [@yangbajing](https://github.com/yangbajing)               | Contributor |          |
+| 刘新元 Liu XinYuan  | [@Liu-XinYuan](https://github.com/Liu-XinYuan)             | Contributor |          |
+| Joker               | [@LiubeyJokerQueue](https://github.com/JokerQueue)         | Contributor | 丰巢     |
+| Eason Lau           | [@Liubey](https://github.com/Liubey)                       | Contributor |          |
+| hailanxin           | [@hailanxin](https://github.com/hailanxin)                 | Contributor |          |
+| Qi Zhang            | [@zzzhangqi](https://github.com/zzzhangqi)                 | Contributor | 好雨科技 |
+| fengxsong           | [@fengxsong](https://github.com/fengxsong)                 | Contributor |          |
+| 谢晓东              | [@Strangevy](https://github.com/Strangevy)                 | Contributor | 花生日记 |
+| ZhaoXinlong         | [@ZhaoXinlong](https://github.com/ZhaoXinlong)             | Contributor |          |
+| xuehaipeng          | [@xuehaipeng](https://github.com/xuehaipeng)               | Contributor |          |
+| 孔令续              | [@mrazkong](https://github.com/mrazkong)                   | Contributor |          |
+| pierre xiong        | [@pierre94](https://github.com/pierre94)                   | Contributor |          |
+| PengShuaixin        | [@PengShuaixin](https://github.com/PengShuaixin)           | Contributor |          |
+| 梁壮                | [@lz](https://github.com/silent-night-no-trace)            | Contributor |          |
+| 张晓寅              | [@ahu0605](https://github.com/ahu0605)                     | Contributor | 电信数智 |
+| 黄海婷              | [@Huanghaiting](https://github.com/Huanghaiting)           | Contributor | 云徙科技 |
+| 任祥德              | [@RenChauncy](https://github.com/RenChauncy)               | Contributor | 探马企服 |
+| 胡圣林              | [@slhu997](https://github.com/slhu997)               | Contributor |  | 
+| 史泽颖              | [@shizeying](https://github.com/shizeying)               | Contributor |  |
+| 王玉博              | [@Wyb7290](https://github.com/Wyb7290)               | Committer | |
+| 伍璇                | [@Luckywustone](https://github.com/Luckywustone)          | Contributor ||
+| 邓苑                | [@CatherineDY](https://github.com/CatherineDY)          | Contributor ||
+| 封琼凤               | [@Luckywustone](https://github.com/fengqiongfeng)          | Committer ||

docs/contribute_guide/贡献指南.md

@@ -0,0 +1,167 @@
+# 贡献指南
+
+- [贡献指南](#贡献指南)
+  - [1、行为准则](#1行为准则)
+  - [2、仓库规范](#2仓库规范)
+    - [2.1、Issue 规范](#21issue-规范)
+    - [2.2、Commit-Log 规范](#22commit-log-规范)
+    - [2.3、Pull-Request 规范](#23pull-request-规范)
+  - [3、操作示例](#3操作示例)
+    - [3.1、初始化环境](#31初始化环境)
+    - [3.2、认领问题](#32认领问题)
+    - [3.3、处理问题 \& 提交解决](#33处理问题--提交解决)
+    - [3.4、请求合并](#34请求合并)
+  - [4、常见问题](#4常见问题)
+    - [4.1、如何将多个 Commit-Log 合并为一个？](#41如何将多个-commit-log-合并为一个)
+
+
+---
+
+
+欢迎 👏🏻 👏🏻 👏🏻 来到 `KnowStreaming`。本文档是关于如何为 `KnowStreaming` 做出贡献的指南。如果您发现不正确或遗漏的内容, 请留下您的意见/建议。
+
+
+---
+
+
+## 1、行为准则
+
+请务必阅读并遵守我们的：[行为准则](https://github.com/didi/KnowStreaming/blob/master/CODE_OF_CONDUCT.md)。
+
+
+## 2、仓库规范
+
+### 2.1、Issue 规范
+
+按要求，在 [创建Issue](https://github.com/didi/KnowStreaming/issues/new/choose) 中创建ISSUE即可。
+
+需要重点说明的是：
+- 提供出现问题的环境信息，包括使用的系统，使用的KS版本等；
+- 提供出现问题的复现方式；
+
+
+### 2.2、Commit-Log 规范
+
+`Commit-Log` 包含三部分 `Header`、`Body`、`Footer`。其中 `Header` 是必须的，格式固定，`Body` 在变更有必要详细解释时使用。
+
+
+**1、`Header` 规范**
+
+`Header` 格式为 `[Type]Message(#IssueID)`， 主要有三部分组成，分别是`Type`、`Message`、`IssueID`，
+
+- `Type`：说明这个提交是哪一个类型的，比如有 Bugfix、Feature、Optimize等；
+- `Message`：说明提交的信息，比如修复xx问题；
+- `IssueID`：该提交，关联的Issue的编号；
+
+
+实际例子：[`[Bugfix]修复新接入的集群，Controller-Host不显示的问题(#927)`](https://github.com/didi/KnowStreaming/pull/933/commits)
+
+
+
+**2、`Body` 规范**
+
+一般不需要，如果解决了较复杂问题，或者代码较多，需要 `Body` 说清楚解决的问题，解决的思路等信息。
+
+---
+
+**3、实际例子**
+
+```
+[Optimize]优化 MySQL & ES 测试容器的初始化(#906)
+
+主要的变更
+1、knowstreaming/knowstreaming-manager 容器；
+2、knowstreaming/knowstreaming-mysql 容器调整为使用 mysql:5.7 容器；
+3、初始化 mysql:5.7 容器后，增加初始化 MySQL 表及数据的动作；
+
+被影响的变更：
+1、移动 km-dist/init/sql 下的MySQL初始化脚本至 km-persistence/src/main/resource/sql 下，以便项目测试时加载到所需的初始化 SQL；
+2、删除无用的 km-dist/init/template 目录；
+3、因为 km-dist/init/sql 和 km-dist/init/template 目录的调整，因此也调整 ReleaseKnowStreaming.xml 内的文件内容；
+```
+
+
+**TODO : 后续有兴趣的同学，可以考虑引入 Git 的 Hook 进行更好的 Commit-Log 的管理。**
+
+
+### 2.3、Pull-Request 规范
+
+详细见：[PULL-REQUEST 模版](../../.github/PULL_REQUEST_TEMPLATE.md)
+
+需要重点说明的是：
+
+- <font color=red > 任何 PR 都必须与有效 ISSUE 相关联。否则， PR 将被拒绝；</font>
+- <font color=red> 一个分支只修改一件事，一个 PR 只修改一件事；</b></font>
+
+---
+
+
+## 3、操作示例
+
+本节主要介绍对 `KnowStreaming` 进行代码贡献时，相关的操作方式及操作命令。
+
+名词说明：
+- 主仓库：https://github.com/didi/KnowStreaming 这个仓库为主仓库。
+- 分仓库：Fork 到自己账号下的 KnowStreaming 仓库为分仓库；
+
+
+### 3.1、初始化环境
+
+1. `Fork KnowStreaming` 主仓库至自己账号下，见 https://github.com/didi/KnowStreaming 地址右上角的 `Fork` 按钮；
+2. 克隆分仓库至本地：`git clone git@github.com:xxxxxxx/KnowStreaming.git`，该仓库的简写名通常是`origin`；
+3. 添加主仓库至本地：`git remote add upstream https://github.com/didi/KnowStreaming`，`upstream`是主仓库在本地的简写名，可以随意命名，前后保持一致即可；
+4. 拉取主仓库代码：`git fetch upstream`；
+5. 拉取分仓库代码：`git fetch origin`；
+6. 将主仓库的`master`分支，拉取到本地并命名为`github_master`：`git checkout -b upstream/master`；
+
+最后，我们来看一下初始化完成之后的大致效果，具体如下图所示：
+![环境初始化](./assets/环境初始化.jpg)
+
+
+至此，我们的环境就初始化好了。后续，`github_master` 分支就是主仓库的`master`分支，我们可以使用`git pull`拉取该分支的最新代码，还可以使用`git checkout -b xxx`拉取我们想要的分支。
+
+
+
+### 3.2、认领问题
+
+在文末评论说明自己要处理该问题即可，具体如下图所示：
+
+![问题认领](./assets/问题认领.jpg)
+
+
+### 3.3、处理问题 & 提交解决
+
+本节主要介绍一下处理问题 & 提交解决过程中的分支管理，具体如下图所示：
+
+![分支管理](./assets/分支管理.png)
+
+1. 切换到主分支：`git checkout github_master`；
+2. 主分支拉最新代码：`git pull`；
+3. 基于主分支拉新分支：`git checkout -b fix_928`；
+4. 提交代码，安装commit的规范进行提交，例如：`git commit -m "[Optimize]优化xxx问题(#928)"`；
+5. 提交到自己远端仓库：`git push --set-upstream origin fix_928`；
+6. `GitHub` 页面发起 `Pull Request` 请求，管理员合入主仓库。这部分详细见下一节；
+
+
+### 3.4、请求合并
+
+代码在提交到 `GitHub` 分仓库之后，就可以在 `GitHub` 的网站创建 `Pull Request`，申请将代码合入主仓库了。 `Pull Request` 具体见下图所示：
+
+![申请合并](./assets/申请合并.jpg)
+
+
+
+[Pull Request 创建的例子](https://github.com/didi/KnowStreaming/pull/945)
+
+
+
+---
+
+
+## 4、常见问题
+
+### 4.1、如何将多个 Commit-Log 合并为一个？
+
+可以使用 `git rebase -i` 命令进行解决。
+
+

docs/dev_guide/支持Kerberos认证的ZK.md

@@ -0,0 +1,69 @@
+
+## 支持Kerberos认证的ZK
+
+
+### 1、修改 KnowStreaming 代码
+
+代码位置：`src/main/java/com/xiaojukeji/know/streaming/km/persistence/kafka/KafkaAdminZKClient.java`
+
+将 `createZKClient` 的 `135行 的 false 改为 true
+![need_modify_code.png](assets/support_kerberos_zk/need_modify_code.png)
+
+
+修改完后重新进行打包编译，打包编译见：[打包编译](https://github.com/didi/KnowStreaming/blob/master/docs/install_guide/%E6%BA%90%E7%A0%81%E7%BC%96%E8%AF%91%E6%89%93%E5%8C%85%E6%89%8B%E5%86%8C.md
+)
+
+
+
+### 2、查看用户在ZK的ACL
+
+假设我们使用的用户是 `kafka` 这个用户。 
+
+- 1、查看 server.properties 的配置的 zookeeper.connect 的地址；
+- 2、使用 `zkCli.sh -serve zookeeper.connect的地址` 登录到ZK页面；
+- 3、ZK页面上，执行命令 `getAcl /kafka` 查看 `kafka` 用户的权限；
+
+此时，我们可以看到如下信息：
+![watch_user_acl.png](assets/support_kerberos_zk/watch_user_acl.png)
+
+`kafka` 用户需要的权限是 `cdrwa`。如果用户没有 `cdrwa` 权限的话，需要创建用户并授权，授权命令为：`setAcl`
+
+
+### 3、创建Kerberos的keytab并修改 KnowStreaming 主机
+
+- 1、在 Kerberos 的域中创建 `kafka/_HOST` 的 `keytab`，并导出。例如：`kafka/dbs-kafka-test-8-53`；
+- 2、导出 keytab 后上传到安装 KS 的机器的 `/etc/keytab` 下；
+- 3、在 KS 机器上，执行 `kinit -kt zookeepe.keytab kafka/dbs-kafka-test-8-53`  看是否能进行 `Kerberos` 登录；
+- 4、可以登录后，配置 `/opt/zookeeper.jaas` 文件，例子如下：
+```sql
+Client {
+    com.sun.security.auth.module.Krb5LoginModule required
+    useKeyTab=true
+    storeKey=false
+    serviceName="zookeeper"
+    keyTab="/etc/keytab/zookeeper.keytab"
+    principal="kafka/dbs-kafka-test-8-53@XXX.XXX.XXX";
+};
+```
+- 5、需要配置 `KDC-Server` 对 `KnowStreaming` 的机器开通防火墙，并在KS的机器 `/etc/host/`  配置 `kdc-server` 的 `hostname`。并将 `krb5.conf` 导入到 `/etc` 下；
+
+
+### 4、修改 KnowStreaming 的配置
+
+- 1、在 `/usr/local/KnowStreaming/KnowStreaming/bin/startup.sh` 中的47行的JAVA_OPT中追加如下设置
+```bash
+-Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/opt/zookeeper.jaas
+```
+
+- 2、重启KS集群后再 start.out 中看到如下信息，则证明Kerberos配置成功；
+
+![success_1.png](assets/support_kerberos_zk/success_1.png)
+
+![success_2.png](assets/support_kerberos_zk/success_2.png)
+
+
+### 5、补充说明
+
+- 1、多Kafka集群如果用的是一样的Kerberos域的话，只需在每个`ZK`中给`kafka`用户配置`crdwa`权限即可，这样集群初始化的时候`zkclient`是都可以认证；
+- 2、当前需要修改代码重新打包才可以支持，后续考虑通过页面支持Kerberos认证的ZK接入；
+- 3、多个Kerberos域暂时未适配；

docs/dev_guide/无数据排查文档.md

@@ -0,0 +1,285 @@
+## 1、集群接入错误
+
+### 1.1、异常现象
+
+如下图所示，集群非空时，大概率为地址配置错误导致。
+
+<img src=http://img-ys011.didistatic.com/static/dc2img/do1_BRiXBvqYFK2dxSF1aqgZ width="80%">
+
+ 
+
+### 1.2、解决方案
+
+接入集群时，依据提示的错误，进行相应的解决。例如：
+
+<img src=http://img-ys011.didistatic.com/static/dc2img/do1_Yn4LhV8aeSEKX1zrrkUi width="50%">
+
+### 1.3、正常情况
+
+接入集群时，页面信息都自动正常出现，没有提示错误。
+
+
+
+## 2、JMX连接失败（需使用3.0.1及以上版本）
+
+### 2.1异常现象
+
+Broker列表的JMX Port列出现红色感叹号，则该Broker的JMX连接异常。
+
+<img src=http://img-ys011.didistatic.com/static/dc2img/do1_MLlLCfAktne4X6MBtBUd width="90%">
+
+
+
+#### 2.1.1、原因一：JMX未开启
+
+##### 2.1.1.1、异常现象
+
+broker列表的JMX Port值为-1，对应Broker的JMX未开启。
+
+<img src=http://img-ys011.didistatic.com/static/dc2img/do1_E1PD8tPsMeR2zYLFBFAu width="90%">
+
+##### 2.1.1.2、解决方案
+
+开启JMX，开启流程如下：
+
+1、修改kafka的bin目录下面的：`kafka-server-start.sh`文件
+
+```
+# 在这个下面增加JMX端口的配置
+if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
+		export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"    
+		export JMX_PORT=9999  # 增加这个配置, 这里的数值并不一定是要9999 
+fi
+```
+
+
+
+2、修改kafka的bin目录下面对的：`kafka-run-class.sh`文件
+
+```
+# JMX settings 
+if [ -z "$KAFKA_JMX_OPTS" ]; then
+	KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false 
+  -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=${当前机器的IP}" 
+fi  
+
+# JMX port to use 
+if [  $JMX_PORT ]; then
+		KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT -				Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT" 
+fi
+```
+
+
+
+3、重启Kafka-Broker。
+
+
+
+#### 2.1.2、原因二：JMX配置错误
+
+##### 2.1.2.1、异常现象
+
+错误日志：
+
+```
+# 错误一： 错误提示的是真实的IP，这样的话基本就是JMX配置的有问题了。 
+2021-01-27 10:06:20.730 ERROR 50901 --- [ics-Thread-1-62] c.x.k.m.c.utils.jmx.JmxConnectorWrap     : JMX connect exception, host:192.168.0.1 port:9999. java.rmi.ConnectException: Connection refused to host: 192.168.0.1; nested exception is:    
+
+# 错误二：错误提示的是127.0.0.1这个IP，这个是机器的hostname配置的可能有问题。 
+2021-01-27 10:06:20.730 ERROR 50901 --- [ics-Thread-1-62] c.x.k.m.c.utils.jmx.JmxConnectorWrap     : JMX connect exception, host:127.0.0.1 port:9999. java.rmi.ConnectException: Connection refused to host: 127.0.0.1;; nested exception is: 
+```
+
+
+
+##### 2.1.2.2、解决方案
+
+开启JMX，开启流程如下：
+
+1、修改kafka的bin目录下面的：`kafka-server-start.sh`文件
+
+```
+# 在这个下面增加JMX端口的配置
+if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
+		export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"    
+		export JMX_PORT=9999  # 增加这个配置, 这里的数值并不一定是要9999 
+fi
+```
+
+
+
+2、修改kafka的bin目录下面对的：`kafka-run-class.sh`文件
+
+```
+# JMX settings 
+if [ -z "$KAFKA_JMX_OPTS" ]; then
+	KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false 
+  -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=${当前机器的IP}" 
+fi  
+
+# JMX port to use 
+if [  $JMX_PORT ]; then
+		KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT -				Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT" 
+fi
+```
+
+
+
+3、重启Kafka-Broker。
+
+
+
+#### 2.1.3、原因三：JMX开启SSL
+
+##### 2.1.3.1、解决方案
+
+<img src=http://img-ys011.didistatic.com/static/dc2img/do1_kNyCi8H9wtHSRkWurB6S width="50%">
+
+#### 2.1.4、原因四：连接了错误IP
+
+##### 2.1.4.1、异常现象
+
+Broker 配置了内外网，而JMX在配置时，可能配置了内网IP或者外网IP，此时`KnowStreaming` 需要连接到特定网络的IP才可以进行访问。
+
+ 比如：Broker在ZK的存储结构如下所示，我们期望连接到 `endpoints` 中标记为 `INTERNAL` 的地址，但是 `KnowStreaming` 却连接了 `EXTERNAL` 的地址。
+
+```json
+{
+    "listener_security_protocol_map": {
+        "EXTERNAL": "SASL_PLAINTEXT",
+        "INTERNAL": "SASL_PLAINTEXT"
+    },
+    "endpoints": [
+        "EXTERNAL://192.168.0.1:7092",
+        "INTERNAL://192.168.0.2:7093"
+    ],
+    "jmx_port": 8099,
+    "host": "192.168.0.1",
+    "timestamp": "1627289710439",
+    "port": -1,
+    "version": 4
+}
+```
+
+##### 2.1.4.2、解决方案
+
+可以手动往`ks_km_physical_cluster`表的`jmx_properties`字段增加一个`useWhichEndpoint`字段，从而控制 `KnowStreaming` 连接到特定的JMX IP及PORT。
+
+`jmx_properties`格式：
+
+```json
+{
+    "maxConn": 100, // KM对单台Broker的最大JMX连接数   
+    "username": "xxxxx", //用户名，可以不填写   
+    "password": "xxxx", // 密码，可以不填写    
+    "openSSL": true, //开启SSL, true表示开启ssl, false表示关闭    
+    "useWhichEndpoint": "EXTERNAL" //指定要连接的网络名称，填写EXTERNAL就是连接endpoints里面的EXTERNAL地址
+}
+```
+
+
+
+SQL例子：
+
+```sql
+UPDATE ks_km_physical_cluster SET jmx_properties='{ "maxConn": 10, "username": "xxxxx", "password": "xxxx", "openSSL": false , "useWhichEndpoint": "xxx"}' where id={xxx};
+```
+
+### 2.2、正常情况
+
+修改完成后，如果看到 JMX PORT这一列全部为绿色，则表示JMX已正常。
+
+<img src=http://img-ys011.didistatic.com/static/dc2img/do1_ymtDTCiDlzfrmSCez2lx width="90%">
+
+
+
+## 3、Elasticsearch问题
+
+注意：mac系统在执行curl指令时，可能报zsh错误。可参考以下操作。
+
+```
+1 进入.zshrc 文件 vim ~/.zshrc 
+2.在.zshrc中加入 setopt no_nomatch 
+3.更新配置 source ~/.zshrc
+```
+
+### 3.1、原因一：缺少索引
+
+#### 3.1.1、异常现象
+
+报错信息
+
+```
+com.didiglobal.logi.elasticsearch.client.model.exception.ESIndexNotFoundException: method [GET], host[http://127.0.0.1:9200], URI [/ks_kafka_broker_metric_2022-10-21,ks_kafka_broker_metric_2022-10-22/_search], status line [HTTP/1.1 404 Not Found]
+```
+
+curl http://{ES的IP地址}:{ES的端口号}/_cat/indices/ks_kafka*  查看KS索引列表，发现没有索引。
+
+#### 3.1.2、解决方案
+
+执行 [ES索引及模版初始化](https://github.com/didi/KnowStreaming/blob/master/bin/init_es_template.sh) 脚本，来创建索引及模版。
+
+
+
+### 3.2、原因二：索引模板错误
+
+#### 3.2.1、异常现象
+
+多集群列表有数据，集群详情页图标无数据。查询KS索引模板列表，发现不存在。
+
+```
+curl {ES的IP地址}:{ES的端口号}/_cat/templates/ks_kafka*?v&h=name 
+```
+
+正常KS模板如下图所示。
+
+<img src=http://img-ys011.didistatic.com/static/dc2img/do1_l79bPYSci9wr6KFwZDA6 width="90%">
+
+
+
+#### 3.2.2、解决方案
+
+删除KS索引模板和索引
+
+```
+curl -XDELETE {ES的IP地址}:{ES的端口号}/ks_kafka*
+curl -XDELETE {ES的IP地址}:{ES的端口号}/_template/ks_kafka*
+```
+
+执行 [ES索引及模版初始化](https://github.com/didi/KnowStreaming/blob/master/bin/init_es_template.sh) 脚本，来创建索引及模版。
+
+
+### 3.3、原因三：集群Shard满
+
+#### 3.3.1、异常现象
+
+报错信息
+
+```
+com.didiglobal.logi.elasticsearch.client.model.exception.ESIndexNotFoundException: method [GET], host[http://127.0.0.1:9200], URI [/ks_kafka_broker_metric_2022-10-21,ks_kafka_broker_metric_2022-10-22/_search], status line [HTTP/1.1 404 Not Found]
+```
+
+尝试手动创建索引失败。
+
+```
+#创建ks_kafka_cluster_metric_test索引的指令
+curl -s  -XPUT http://{ES的IP地址}:{ES的端口号}/ks_kafka_cluster_metric_test
+```
+
+#### 3.3.2、解决方案
+
+ES索引的默认分片数量为1000，达到数量以后，索引创建失败。
+
++ 扩大ES索引数量上限，执行指令
+
+```
+curl -XPUT  -H"content-type:application/json" http://{ES的IP地址}:{ES的端口号}/_cluster/settings -d '
+{
+  "persistent": {
+    "cluster": {
+      "max_shards_per_node":{索引上限，默认为1000}
+    }
+  }
+}'
+```
+
+执行 [ES索引及模版初始化](https://github.com/didi/KnowStreaming/blob/master/bin/init_es_template.sh) 脚本，来补全索引。

docs/install_guide/单机部署手册.md

@@ -74,11 +74,11 @@ sh deploy_KnowStreaming-offline.sh
 ```bash
 # 相关镜像在Docker Hub都可以下载
 # 快速安装(NAMESPACE需要更改为已存在的，安装启动需要几分钟初始化请稍等~)
-helm install -n [NAMESPACE] [NAME] http://download.knowstreaming.com/charts/knowstreaming-manager-0.1.3.tgz
+helm install -n [NAMESPACE] [NAME] http://download.knowstreaming.com/charts/knowstreaming-manager-0.1.5.tgz
 
 # 获取KnowStreaming前端ui的service. 默认nodeport方式.
 # (http://nodeIP:nodeport，默认用户名密码：admin/admin2022_)
-# `v3.0.0-beta.2`版本开始，默认账号密码为`admin` / `admin`；
+# `v3.0.0-beta.2`版本开始（helm chart包版本0.1.4开始），默认账号密码为`admin` / `admin`；
 
 # 添加仓库
 helm repo add knowstreaming http://download.knowstreaming.com/charts
@@ -90,13 +90,54 @@ helm pull knowstreaming/knowstreaming-manager
  
 
 #### 2.1.3.2、Docker Compose
-```yml
+**环境依赖**
-version: "3"
 
+- [Docker](https://docs.docker.com/engine/install/) 
+- [Docker Compose](https://docs.docker.com/compose/install/) 
+
+
+**安装命令**
+```bash
+# `v3.0.0-beta.2`版本开始(docker镜像为0.2.0版本开始)，默认账号密码为`admin` / `admin`；
+# https://hub.docker.com/u/knowstreaming 在此处寻找最新镜像版本
+# mysql与es可以使用自己搭建的服务,调整对应配置即可
+
+# 复制docker-compose.yml到指定位置后执行下方命令即可启动
+docker-compose up -d
+```
+
+**验证安装**
+```shell
+docker-compose ps
+# 验证启动 - 状态为 UP 则表示成功 
+        Name                       Command                       State                  Ports
+----------------------------------------------------------------------------------------------------
+elasticsearch-single    /usr/local/bin/docker-entr ...   Up                      9200/tcp, 9300/tcp
+knowstreaming-init      /bin/bash /es_template_cre ...   Up
+knowstreaming-manager   /bin/sh /ks-start.sh             Up                      80/tcp
+knowstreaming-mysql     /entrypoint.sh mysqld            Up (health: starting)   3306/tcp, 33060/tcp
+knowstreaming-ui        /docker-entrypoint.sh ngin ...   Up                      0.0.0.0:80->80/tcp
+
+# 稍等一分钟左右 knowstreaming-init 会退出，表示es初始化完成，可以访问页面
+        Name                       Command                  State              Ports
+-------------------------------------------------------------------------------------------
+knowstreaming-init      /bin/bash /es_template_cre ...   Exit 0
+knowstreaming-mysql     /entrypoint.sh mysqld            Up (healthy)   3306/tcp, 33060/tcp
+```
+
+**访问**
+```http request
+http://127.0.0.1:80/
+```
+
+
+**docker-compose.yml**
+```yml
+version: "2"
 services:
-  
+  # *不要调整knowstreaming-manager服务名称，ui中会用到
   knowstreaming-manager:
-    image: knowstreaming/knowstreaming-manager:0.2.0-test
+    image: knowstreaming/knowstreaming-manager:latest
     container_name: knowstreaming-manager
     privileged: true
     restart: always
@@ -110,33 +151,36 @@ services:
       - /ks-start.sh
     environment:
       TZ: Asia/Shanghai
-
+      # mysql服务地址
       SERVER_MYSQL_ADDRESS: knowstreaming-mysql:3306
+      # mysql数据库名
       SERVER_MYSQL_DB: know_streaming
+      # mysql用户名
       SERVER_MYSQL_USER: root
+      # mysql用户密码
       SERVER_MYSQL_PASSWORD: admin2022_
-
+      # es服务地址
       SERVER_ES_ADDRESS: elasticsearch-single:9200
-
+      # 服务JVM参数
       JAVA_OPTS: -Xmx1g -Xms1g
-
+      # 对于kafka中ADVERTISED_LISTENERS填写的hostname可以通过该方式完成
 #    extra_hosts:
 #      - "hostname:x.x.x.x"
+      # 服务日志路径
 #    volumes:
 #      - /ks/manage/log:/logs
   knowstreaming-ui:
-    image: knowstreaming/knowstreaming-ui:0.2.0-test1
+    image: knowstreaming/knowstreaming-ui:latest
     container_name: knowstreaming-ui
     restart: always
     ports:
-      - '18092:80'
+      - '80:80'
     environment:
       TZ: Asia/Shanghai
     depends_on:
       - knowstreaming-manager
 #    extra_hosts:
 #      - "hostname:x.x.x.x"
-
   elasticsearch-single:
     image: docker.io/library/elasticsearch:7.6.2
     container_name: elasticsearch-single
@@ -149,14 +193,19 @@ services:
 #      - '9300:9300'
     environment:
       TZ: Asia/Shanghai
+      # es的JVM参数
       ES_JAVA_OPTS: -Xms512m -Xmx512m
+      # 单节点配置，多节点集群参考 https://www.elastic.co/guide/en/elasticsearch/reference/7.6/docker.html#docker-compose-file
       discovery.type: single-node
+      # 数据持久化路径
 #    volumes:
 #      - /ks/es/data:/usr/share/elasticsearch/data
 
+  # es初始化服务，与manager使用同一镜像
+  # 首次启动es需初始化模版和索引,后续会自动创建
   knowstreaming-init:
-    image: knowstreaming/knowstreaming-manager:0.2.0-test
+    image: knowstreaming/knowstreaming-manager:latest
-    container_name: knowstreaming_init
+    container_name: knowstreaming-init
     depends_on:
       - elasticsearch-single
     command:
@@ -164,22 +213,26 @@ services:
       - /es_template_create.sh
     environment:
       TZ: Asia/Shanghai
+      # es服务地址
       SERVER_ES_ADDRESS: elasticsearch-single:9200
 
-
   knowstreaming-mysql:
-    image: knowstreaming/knowstreaming-mysql:0.2.0-test
+    image: knowstreaming/knowstreaming-mysql:latest
     container_name: knowstreaming-mysql
     restart: always
     environment:
       TZ: Asia/Shanghai
+      # root 用户密码
       MYSQL_ROOT_PASSWORD: admin2022_
+      # 初始化时创建的数据库名称
       MYSQL_DATABASE: know_streaming
+      # 通配所有host,可以访问远程
       MYSQL_ROOT_HOST: '%'
     expose:
       - 3306
 #    ports:
 #      - '3306:3306'
+      # 数据持久化路径
 #    volumes:
 #      - /ks/mysql/data:/data/mysql
 ```

docs/install_guide/版本升级手册.md

@@ -1,12 +1,342 @@
 ## 6.2、版本升级手册
 
-注意：如果想升级至具体版本，需要将你当前版本至你期望使用版本的变更统统执行一遍，然后才能正常使用。
+注意：
+- 如果想升级至具体版本，需要将你当前版本至你期望使用版本的变更统统执行一遍，然后才能正常使用。
+- 如果中间某个版本没有升级信息，则表示该版本直接替换安装包即可从前一个版本升级至当前版本。
 
-### 6.2.0、升级至 `master` 版本
+### 升级至 `master` 版本
 
-暂无
 
-### 6.2.1、升级至 `v3.0.0-beta.2`版本
+### 升级至 `3.3.0` 版本
+
+**SQL 变更**
+```sql
+ALTER TABLE `logi_security_user`
+    CHANGE COLUMN `phone` `phone` VARCHAR(20) NOT NULL DEFAULT '' COMMENT 'mobile' ;
+
+ALTER TABLE ks_kc_connector ADD `heartbeat_connector_name` varchar(512) DEFAULT '' COMMENT '心跳检测connector名称';
+ALTER TABLE ks_kc_connector ADD `checkpoint_connector_name` varchar(512) DEFAULT '' COMMENT '进度确认connector名称';
+
+INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_MIRROR_MAKER_TOTAL_RECORD_ERRORS', '{\"value\" : 1}', 'MirrorMaker消息处理错误的次数', 'admin');
+INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_MIRROR_MAKER_REPLICATION_LATENCY_MS_MAX', '{\"value\" : 6000}', 'MirrorMaker消息复制最大延迟时间', 'admin');
+INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_MIRROR_MAKER_UNASSIGNED_TASK_COUNT', '{\"value\" : 20}', 'MirrorMaker未被分配的任务数量', 'admin');
+INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_MIRROR_MAKER_FAILED_TASK_COUNT', '{\"value\" : 10}', 'MirrorMaker失败状态的任务数量', 'admin');
+
+
+-- 多集群管理权限2023-01-05新增
+INSERT INTO `logi_security_permission` (`id`, `permission_name`, `parent_id`, `leaf`, `level`, `description`, `is_delete`, `app_name`) VALUES ('2012', 'Topic-新增Topic复制', '1593', '1', '2', 'Topic-新增Topic复制', '0', 'know-streaming');
+INSERT INTO `logi_security_permission` (`id`, `permission_name`, `parent_id`, `leaf`, `level`, `description`, `is_delete`, `app_name`) VALUES ('2014', 'Topic-详情-取消Topic复制', '1593', '1', '2', 'Topic-详情-取消Topic复制', '0', 'know-streaming');
+
+INSERT INTO `logi_security_role_permission` (`role_id`, `permission_id`, `is_delete`, `app_name`) VALUES ('1677', '2012', '0', 'know-streaming');
+INSERT INTO `logi_security_role_permission` (`role_id`, `permission_id`, `is_delete`, `app_name`) VALUES ('1677', '2014', '0', 'know-streaming');
+
+
+-- 多集群管理权限2023-01-18新增
+INSERT INTO `logi_security_permission` (`id`, `permission_name`, `parent_id`, `leaf`, `level`, `description`, `is_delete`, `app_name`) VALUES ('2016', 'MM2-新增', '1593', '1', '2', 'MM2-新增', '0', 'know-streaming');
+INSERT INTO `logi_security_permission` (`id`, `permission_name`, `parent_id`, `leaf`, `level`, `description`, `is_delete`, `app_name`) VALUES ('2018', 'MM2-编辑', '1593', '1', '2', 'MM2-编辑', '0', 'know-streaming');
+INSERT INTO `logi_security_permission` (`id`, `permission_name`, `parent_id`, `leaf`, `level`, `description`, `is_delete`, `app_name`) VALUES ('2020', 'MM2-删除', '1593', '1', '2', 'MM2-删除', '0', 'know-streaming');
+INSERT INTO `logi_security_permission` (`id`, `permission_name`, `parent_id`, `leaf`, `level`, `description`, `is_delete`, `app_name`) VALUES ('2022', 'MM2-重启', '1593', '1', '2', 'MM2-重启', '0', 'know-streaming');
+INSERT INTO `logi_security_permission` (`id`, `permission_name`, `parent_id`, `leaf`, `level`, `description`, `is_delete`, `app_name`) VALUES ('2024', 'MM2-暂停&恢复', '1593', '1', '2', 'MM2-暂停&恢复', '0', 'know-streaming');
+
+INSERT INTO `logi_security_role_permission` (`role_id`, `permission_id`, `is_delete`, `app_name`) VALUES ('1677', '2016', '0', 'know-streaming');
+INSERT INTO `logi_security_role_permission` (`role_id`, `permission_id`, `is_delete`, `app_name`) VALUES ('1677', '2018', '0', 'know-streaming');
+INSERT INTO `logi_security_role_permission` (`role_id`, `permission_id`, `is_delete`, `app_name`) VALUES ('1677', '2020', '0', 'know-streaming');
+INSERT INTO `logi_security_role_permission` (`role_id`, `permission_id`, `is_delete`, `app_name`) VALUES ('1677', '2022', '0', 'know-streaming');
+INSERT INTO `logi_security_role_permission` (`role_id`, `permission_id`, `is_delete`, `app_name`) VALUES ('1677', '2024', '0', 'know-streaming');
+
+
+DROP TABLE IF EXISTS `ks_ha_active_standby_relation`;
+CREATE TABLE `ks_ha_active_standby_relation` (
+    `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
+    `active_cluster_phy_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT '主集群ID',
+    `standby_cluster_phy_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT '备集群ID',
+    `res_name` varchar(192) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT '资源名称',
+    `res_type` int(11) NOT NULL DEFAULT '-1' COMMENT '资源类型，0：集群，1：镜像Topic，2：主备Topic',
+    `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
+    `modify_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
+    `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
+    PRIMARY KEY (`id`),
+    UNIQUE KEY `uniq_cluster_res` (`res_type`,`active_cluster_phy_id`,`standby_cluster_phy_id`,`res_name`),
+    UNIQUE KEY `uniq_res_type_standby_cluster_res_name` (`res_type`,`standby_cluster_phy_id`,`res_name`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='HA主备关系表';
+    
+
+-- 删除idx_cluster_phy_id 索引并新增idx_cluster_update_time索引
+ALTER TABLE `ks_km_kafka_change_record` DROP INDEX `idx_cluster_phy_id` ,
+ADD INDEX `idx_cluster_update_time` (`cluster_phy_id` ASC, `update_time` ASC);
+```
+
+### 升级至 `3.2.0` 版本
+
+**配置变更**
+
+```yaml
+# 新增如下配置
+
+spring:
+  logi-job:  # know-streaming 依赖的 logi-job 模块的数据库的配置，默认与 know-streaming 的数据库配置保持一致即可
+    enable: true                # true表示开启job任务, false表关闭。KS在部署上可以考虑部署两套服务，一套处理前端请求，一套执行job任务，此时可以通过该字段进行控制
+
+# 线程池大小相关配置
+thread-pool:
+  es:
+    search:                         # es查询线程池
+      thread-num: 20                # 线程池大小
+      queue-size: 10000             # 队列大小
+
+# 客户端池大小相关配置
+client-pool:
+  kafka-admin:
+    client-cnt: 1                 # 每个Kafka集群创建的KafkaAdminClient数
+
+# ES客户端配置
+es:
+  index:
+    expire: 15                    # 索引过期天数，15表示超过15天的索引会被KS过期删除
+```
+
+**SQL 变更**
+```sql
+DROP TABLE IF EXISTS `ks_kc_connect_cluster`;
+CREATE TABLE `ks_kc_connect_cluster` (
+     `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'Connect集群ID',
+     `kafka_cluster_phy_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT 'Kafka集群ID',
+     `name` varchar(128) NOT NULL DEFAULT '' COMMENT '集群名称',
+     `group_name` varchar(128) NOT NULL DEFAULT '' COMMENT '集群Group名称',
+     `cluster_url` varchar(1024) NOT NULL DEFAULT '' COMMENT '集群地址',
+     `member_leader_url` varchar(1024) NOT NULL DEFAULT '' COMMENT 'URL地址',
+     `version` varchar(64) NOT NULL DEFAULT '' COMMENT 'connect版本',
+     `jmx_properties` text COMMENT 'JMX配置',
+     `state` tinyint(4) NOT NULL DEFAULT '1' COMMENT '集群使用的消费组状态，也表示集群状态:-1 Unknown,0 ReBalance,1 Active,2 Dead,3 Empty',
+     `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '接入时间',
+     `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
+     PRIMARY KEY (`id`),
+     UNIQUE KEY `uniq_id_group_name` (`id`,`group_name`),
+     UNIQUE KEY `uniq_name_kafka_cluster` (`name`,`kafka_cluster_phy_id`),
+     KEY `idx_kafka_cluster_phy_id` (`kafka_cluster_phy_id`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Connect集群信息表';
+
+
+DROP TABLE IF EXISTS `ks_kc_connector`;
+CREATE TABLE `ks_kc_connector` (
+     `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
+     `kafka_cluster_phy_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT 'Kafka集群ID',
+     `connect_cluster_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT 'Connect集群ID',
+     `connector_name` varchar(512) NOT NULL DEFAULT '' COMMENT 'Connector名称',
+     `connector_class_name` varchar(512) NOT NULL DEFAULT '' COMMENT 'Connector类',
+     `connector_type` varchar(32) NOT NULL DEFAULT '' COMMENT 'Connector类型',
+     `state` varchar(45) NOT NULL DEFAULT '' COMMENT '状态',
+     `topics` text COMMENT '访问过的Topics',
+     `task_count` int(11) NOT NULL DEFAULT '0' COMMENT '任务数',
+     `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
+     `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
+     PRIMARY KEY (`id`),
+     UNIQUE KEY `uniq_connect_cluster_id_connector_name` (`connect_cluster_id`,`connector_name`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Connector信息表';
+
+
+DROP TABLE IF EXISTS `ks_kc_worker`;
+CREATE TABLE `ks_kc_worker` (
+     `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
+     `kafka_cluster_phy_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT 'Kafka集群ID',
+     `connect_cluster_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT 'Connect集群ID',
+     `member_id` varchar(512) NOT NULL DEFAULT '' COMMENT '成员ID',
+     `host` varchar(128) NOT NULL DEFAULT '' COMMENT '主机名',
+     `jmx_port` int(16) NOT NULL DEFAULT '-1' COMMENT 'Jmx端口',
+     `url` varchar(1024) NOT NULL DEFAULT '' COMMENT 'URL信息',
+     `leader_url` varchar(1024) NOT NULL DEFAULT '' COMMENT 'leaderURL信息',
+     `leader` int(16) NOT NULL DEFAULT '0' COMMENT '状态: 1是leader，0不是leader',
+     `worker_id` varchar(128) NOT NULL COMMENT 'worker地址',
+     `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
+     `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
+     PRIMARY KEY (`id`),
+     UNIQUE KEY `uniq_cluster_id_member_id` (`connect_cluster_id`,`member_id`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='worker信息表';
+
+
+DROP TABLE IF EXISTS `ks_kc_worker_connector`;
+CREATE TABLE `ks_kc_worker_connector` (
+     `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
+     `kafka_cluster_phy_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT 'Kafka集群ID',
+     `connect_cluster_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT 'Connect集群ID',
+     `connector_name` varchar(512) NOT NULL DEFAULT '' COMMENT 'Connector名称',
+     `worker_member_id` varchar(256) NOT NULL DEFAULT '',
+     `task_id` int(16) NOT NULL DEFAULT '-1' COMMENT 'Task的ID',
+     `state` varchar(128) DEFAULT NULL COMMENT '任务状态',
+     `worker_id` varchar(128) DEFAULT NULL COMMENT 'worker信息',
+     `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
+     `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
+     PRIMARY KEY (`id`),
+     UNIQUE KEY `uniq_relation` (`connect_cluster_id`,`connector_name`,`task_id`,`worker_member_id`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Worker和Connector关系表';
+
+INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_CONNECTOR_FAILED_TASK_COUNT', '{\"value\" : 1}', 'connector失败状态的任务数量', 'admin');
+INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_CONNECTOR_UNASSIGNED_TASK_COUNT', '{\"value\" : 1}', 'connector未被分配的任务数量', 'admin');
+INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_CONNECT_CLUSTER_TASK_STARTUP_FAILURE_PERCENTAGE', '{\"value\" : 0.05}', 'Connect集群任务启动失败概率', 'admin');
+```
+
+---
+
+### 升级至 `v3.1.0` 版本
+
+```sql
+INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_ZK_BRAIN_SPLIT', '{ 	\"value\": 1} ', 'ZK 脑裂', 'admin');
+INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_ZK_OUTSTANDING_REQUESTS', '{ 	\"amount\": 100, \"ratio\":0.8} ', 'ZK Outstanding 请求堆积数', 'admin');
+INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_ZK_WATCH_COUNT', '{ 	\"amount\": 100000, 	\"ratio\": 0.8 } ', 'ZK WatchCount 数', 'admin');
+INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_ZK_ALIVE_CONNECTIONS', '{ 	\"amount\": 10000, 	\"ratio\": 0.8 } ', 'ZK 连接数', 'admin');
+INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_ZK_APPROXIMATE_DATA_SIZE', '{ 	\"amount\": 524288000, 	\"ratio\": 0.8 } ', 'ZK 数据大小(Byte)', 'admin');
+INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_ZK_SENT_RATE', '{ 	\"amount\": 500000, 	\"ratio\": 0.8 } ', 'ZK 发包数', 'admin');
+
+```
+
+### 升级至 `v3.0.1` 版本
+
+**ES 索引模版**
+```bash
+# 新增 ks_kafka_zookeeper_metric 索引模版。
+# 可通过再次执行 bin/init_es_template.sh 脚本，创建该索引模版。
+
+# 索引模版内容
+PUT _template/ks_kafka_zookeeper_metric
+{
+    "order" : 10,
+    "index_patterns" : [
+      "ks_kafka_zookeeper_metric*"
+    ],
+    "settings" : {
+      "index" : {
+        "number_of_shards" : "10"
+      }
+    },
+    "mappings" : {
+      "properties" : {
+        "routingValue" : {
+          "type" : "text",
+          "fields" : {
+            "keyword" : {
+              "ignore_above" : 256,
+              "type" : "keyword"
+            }
+          }
+        },
+        "clusterPhyId" : {
+          "type" : "long"
+        },
+        "metrics" : {
+          "properties" : {
+            "AvgRequestLatency" : {
+              "type" : "double"
+            },
+            "MinRequestLatency" : {
+              "type" : "double"
+            },
+            "MaxRequestLatency" : {
+              "type" : "double"
+            },
+            "OutstandingRequests" : {
+              "type" : "double"
+            },
+            "NodeCount" : {
+              "type" : "double"
+            },
+            "WatchCount" : {
+              "type" : "double"
+            },
+            "NumAliveConnections" : {
+              "type" : "double"
+            },
+            "PacketsReceived" : {
+              "type" : "double"
+            },
+            "PacketsSent" : {
+              "type" : "double"
+            },
+            "EphemeralsCount" : {
+              "type" : "double"
+            },
+            "ApproximateDataSize" : {
+              "type" : "double"
+            },
+            "OpenFileDescriptorCount" : {
+              "type" : "double"
+            },
+            "MaxFileDescriptorCount" : {
+              "type" : "double"
+            }
+          }
+        },
+        "key" : {
+          "type" : "text",
+          "fields" : {
+            "keyword" : {
+              "ignore_above" : 256,
+              "type" : "keyword"
+            }
+          }
+        },
+        "timestamp" : {
+          "format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
+          "type" : "date"
+        }
+      }
+    },
+    "aliases" : { }
+  }
+```
+
+
+**SQL 变更**
+
+```sql
+DROP TABLE IF EXISTS `ks_km_zookeeper`;
+CREATE TABLE `ks_km_zookeeper` (
+    `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
+    `cluster_phy_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT '物理集群ID',
+    `host` varchar(128) NOT NULL DEFAULT '' COMMENT 'zookeeper主机名',
+    `port` int(16) NOT NULL DEFAULT '-1' COMMENT 'zookeeper端口',
+    `role` varchar(16) NOT NULL DEFAULT '' COMMENT '角色, leader follower observer',
+    `version` varchar(128) NOT NULL DEFAULT '' COMMENT 'zookeeper版本',
+    `status` int(16) NOT NULL DEFAULT '0' COMMENT '状态: 1存活，0未存活，11存活但是4字命令使用不了',
+    `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
+    `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
+    PRIMARY KEY (`id`),
+    UNIQUE KEY `uniq_cluster_phy_id_host_port` (`cluster_phy_id`,`host`, `port`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Zookeeper信息表';
+
+
+DROP TABLE IF EXISTS `ks_km_group`;
+CREATE TABLE `ks_km_group` (
+     `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
+     `cluster_phy_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT '集群id',
+     `name` varchar(192) COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT 'Group名称',
+     `member_count` int(11) unsigned NOT NULL DEFAULT '0' COMMENT '成员数',
+     `topic_members` text CHARACTER SET utf8 COMMENT 'group消费的topic列表',
+     `partition_assignor` varchar(255) CHARACTER SET utf8 NOT NULL COMMENT '分配策略',
+     `coordinator_id` int(11) NOT NULL COMMENT 'group协调器brokerId',
+     `type` int(11) NOT NULL COMMENT 'group类型 0：consumer 1：connector',
+     `state` varchar(64) CHARACTER SET utf8 NOT NULL DEFAULT '' COMMENT '状态',
+     `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
+     `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
+    PRIMARY KEY (`id`),
+    UNIQUE KEY `uniq_cluster_phy_id_name` (`cluster_phy_id`,`name`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Group信息表';
+
+```
+
+
+### 升级至 `v3.0.0` 版本
+
+**SQL 变更**
+
+```sql
+ALTER TABLE `ks_km_physical_cluster` 
+ADD COLUMN `zk_properties` TEXT NULL COMMENT 'ZK配置' AFTER `jmx_properties`;
+```
+
+---
+
+
+### 升级至 `v3.0.0-beta.2`版本
 
 **配置变更**
 
@@ -77,7 +407,7 @@ ALTER TABLE `logi_security_oplog`
 
 ---
 
-### 6.2.2、升级至 `v3.0.0-beta.1`版本
+### 升级至 `v3.0.0-beta.1`版本
 
 **SQL 变更**
 
@@ -96,7 +426,7 @@ ALTER COLUMN `operation_methods` set default '';
 
 ---
 
-### 6.2.3、`2.x`版本 升级至 `v3.0.0-beta.0`版本
+### `2.x`版本 升级至 `v3.0.0-beta.0`版本
 
 **升级步骤：**
 

docs/user_guide/faq.md

@@ -37,7 +37,7 @@
 
 ## 8.4、`Jmx`连接失败如何解决？
 
-- 参看 [Jmx 连接配置&问题解决](./9-attachment#jmx-连接失败问题解决) 说明。
+- 参看 [Jmx 连接配置&问题解决](https://doc.knowstreaming.com/product/9-attachment#91jmx-%E8%BF%9E%E6%8E%A5%E5%A4%B1%E8%B4%A5%E9%97%AE%E9%A2%98%E8%A7%A3%E5%86%B3) 说明。
 
  
 
@@ -182,3 +182,47 @@ Node 版本: v12.22.12
 
 + 原因：由于数据库编码和我们提供的脚本不一致，数据库里的数据发生了乱码，因此出现权限识别失败问题。
 + 解决方案：清空数据库数据，将数据库字符集调整为utf8，最后重新执行[dml-logi.sql](https://github.com/didi/KnowStreaming/blob/master/km-dist/init/sql/dml-logi.sql)脚本导入数据即可。
+
+
+## 8.13、接入开启kerberos认证的kafka集群
+
+1. 部署KnowStreaming的机器上安装krb客户端；
+2. 替换/etc/krb5.conf配置文件；
+3. 把kafka对应的keytab复制到改机器目录下；
+4. 接入集群时认证配置，配置信息根据实际情况填写；
+```json
+{
+  "security.protocol": "SASL_PLAINTEXT",
+  "sasl.mechanism": "GSSAPI",
+  "sasl.jaas.config": "com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab=\"/etc/keytab/kafka.keytab\" storeKey=true useTicketCache=false principal=\"kafka/kafka@TEST.COM\";",
+  "sasl.kerberos.service.name": "kafka"
+}
+```
+
+
+## 8.14、对接Ldap的配置
+
+```yaml
+# 需要在application.yml中增加如下配置。相关配置的信息，按实际情况进行调整
+account:
+  ldap:
+    url: ldap://127.0.0.1:8080/
+    basedn: DC=senz,DC=local
+    factory: com.sun.jndi.ldap.LdapCtxFactory
+    filter: sAMAccountName
+    security:
+      authentication: simple
+      principal: CN=search,DC=senz,DC=local
+      credentials: xxxxxxx
+    auth-user-registration: false # 是否注册到mysql，默认false
+    auth-user-registration-role: 1677 # 1677是超级管理员角色的id，如果赋予想默认赋予普通角色，可以到ks新建一个。
+
+# 需要在application.yml中修改如下配置
+spring:
+  logi-security:
+    login-extend-bean-name: ksLdapLoginService # 表示使用ldap的service
+```
+
+## 8.15、测试时使用Testcontainers的说明
+1. 需要docker运行环境 [Testcontainers运行环境说明](https://www.testcontainers.org/supported_docker_environment/)
+2. 如果本机没有docker，可以使用[远程访问docker](https://docs.docker.com/config/daemon/remote-access/) [Testcontainers配置说明](https://www.testcontainers.org/features/configuration/#customizing-docker-host-detection)  

km-biz/pom.xml

@@ -29,6 +29,11 @@
             <artifactId>km-core</artifactId>
             <version>${project.parent.version}</version>
         </dependency>
+        <dependency>
+            <groupId>com.xiaojukeji.kafka</groupId>
+            <artifactId>km-rebalance</artifactId>
+            <version>${project.parent.version}</version>
+        </dependency>
 
         <!-- spring -->
         <dependency>
@@ -62,10 +67,6 @@
             <groupId>commons-lang</groupId>
             <artifactId>commons-lang</artifactId>
         </dependency>
-        <dependency>
-            <groupId>junit</groupId>
-            <artifactId>junit</artifactId>
-        </dependency>
 
         <dependency>
             <groupId>commons-codec</groupId>

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/cluster/ClusterConnectorsManager.java

@@ -0,0 +1,15 @@
+package com.xiaojukeji.know.streaming.km.biz.cluster;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.ClusterConnectorsOverviewDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.connect.ConnectStateVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.connector.ClusterConnectorOverviewVO;
+
+/**
+ * Kafka集群Connector概览
+ */
+public interface ClusterConnectorsManager {
+    PaginationResult<ClusterConnectorOverviewVO> getClusterConnectorsOverview(Long clusterPhyId, ClusterConnectorsOverviewDTO dto);
+
+    ConnectStateVO getClusterConnectorsState(Long clusterPhyId);
+}

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/cluster/ClusterZookeepersManager.java

@@ -0,0 +1,19 @@
+package com.xiaojukeji.know.streaming.km.biz.cluster;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.ClusterZookeepersOverviewDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.zookeeper.ClusterZookeepersOverviewVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.zookeeper.ClusterZookeepersStateVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.zookeeper.ZnodeVO;
+
+/**
+ * 多集群总体状态
+ */
+public interface ClusterZookeepersManager {
+    Result<ClusterZookeepersStateVO> getClusterPhyZookeepersState(Long clusterPhyId);
+
+    PaginationResult<ClusterZookeepersOverviewVO> getClusterPhyZookeepersOverview(Long clusterPhyId, ClusterZookeepersOverviewDTO dto);
+
+    Result<ZnodeVO> getZnodeVO(Long clusterPhyId, String path);
+}

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/cluster/MultiClusterPhyManager.java

@@ -1,10 +1,15 @@
 package com.xiaojukeji.know.streaming.km.biz.cluster;
 
+import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhysHealthState;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhysState;
 import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.MultiClusterDashboardDTO;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.ClusterPhyBaseVO;
 import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.ClusterPhyDashboardVO;
 
+import java.util.List;
+
 /**
  * 多集群总体状态
  */
@@ -15,10 +20,14 @@ public interface MultiClusterPhyManager {
      */
     ClusterPhysState getClusterPhysState();
 
+    ClusterPhysHealthState getClusterPhysHealthState();
+
     /**
      * 查询多集群大盘
      * @param dto 分页信息
      * @return
      */
     PaginationResult<ClusterPhyDashboardVO> getClusterPhysDashboard(MultiClusterDashboardDTO dto);
+
+    Result<List<ClusterPhyBaseVO>> getClusterPhysBasic();
 }

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/cluster/impl/ClusterBrokersManagerImpl.java

@@ -6,6 +6,8 @@ import com.xiaojukeji.know.streaming.km.biz.cluster.ClusterBrokersManager;
 import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.ClusterBrokersOverviewDTO;
 import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationSortDTO;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.broker.Broker;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.config.JmxConfig;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.kafkacontroller.KafkaController;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.BrokerMetrics;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
@@ -16,6 +18,8 @@ import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.res.ClusterBroker
 import com.xiaojukeji.know.streaming.km.common.bean.vo.kafkacontroller.KafkaControllerVO;
 import com.xiaojukeji.know.streaming.km.common.constant.KafkaConstant;
 import com.xiaojukeji.know.streaming.km.common.enums.SortTypeEnum;
+import com.xiaojukeji.know.streaming.km.common.enums.cluster.ClusterRunStateEnum;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.PaginationMetricsUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.PaginationUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
@@ -24,6 +28,8 @@ import com.xiaojukeji.know.streaming.km.core.service.broker.BrokerMetricService;
 import com.xiaojukeji.know.streaming.km.core.service.broker.BrokerService;
 import com.xiaojukeji.know.streaming.km.core.service.kafkacontroller.KafkaControllerService;
 import com.xiaojukeji.know.streaming.km.core.service.topic.TopicService;
+import com.xiaojukeji.know.streaming.km.persistence.cache.LoadedClusterPhyCache;
+import com.xiaojukeji.know.streaming.km.persistence.kafka.KafkaJMXClient;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Service;
 
@@ -51,6 +57,9 @@ public class ClusterBrokersManagerImpl implements ClusterBrokersManager {
     @Autowired
     private KafkaControllerService kafkaControllerService;
 
+    @Autowired
+    private KafkaJMXClient kafkaJMXClient;
+
     @Override
     public PaginationResult<ClusterBrokersOverviewVO> getClusterPhyBrokersOverview(Long clusterPhyId, ClusterBrokersOverviewDTO dto) {
         // 获取集群Broker列表
@@ -75,15 +84,24 @@ public class ClusterBrokersManagerImpl implements ClusterBrokersManager {
         //获取controller信息
         KafkaController kafkaController = kafkaControllerService.getKafkaControllerFromDB(clusterPhyId);
 
+        //获取jmx状态信息
+        Map<Integer, Boolean> jmxConnectedMap = new HashMap<>();
+        brokerList.forEach(elem -> jmxConnectedMap.put(elem.getBrokerId(), kafkaJMXClient.getClientWithCheck(clusterPhyId, elem.getBrokerId()) != null));
+
+
+        ClusterPhy clusterPhy = LoadedClusterPhyCache.getByPhyId(clusterPhyId);
+
         // 格式转换
         return PaginationResult.buildSuc(
                 this.convert2ClusterBrokersOverviewVOList(
+                        clusterPhy,
                         paginationResult.getData().getBizData(),
                         brokerList,
                         metricsResult.getData(),
                         groupTopic,
                         transactionTopic,
-                        kafkaController
+                        kafkaController,
+                        jmxConnectedMap
                 ),
                 paginationResult
         );
@@ -122,7 +140,8 @@ public class ClusterBrokersManagerImpl implements ClusterBrokersManager {
             clusterBrokersStateVO.setKafkaControllerAlive(true);
         }
 
-        clusterBrokersStateVO.setConfigSimilar(brokerConfigService.countBrokerConfigDiffsFromDB(clusterPhyId, Arrays.asList("broker.id", "listeners", "name", "value")) <= 0);
+        clusterBrokersStateVO.setConfigSimilar(brokerConfigService.countBrokerConfigDiffsFromDB(clusterPhyId, KafkaConstant.CONFIG_SIMILAR_IGNORED_CONFIG_KEY_LIST) <= 0
+        );
 
         return clusterBrokersStateVO;
     }
@@ -160,27 +179,36 @@ public class ClusterBrokersManagerImpl implements ClusterBrokersManager {
         );
     }
 
-    private List<ClusterBrokersOverviewVO> convert2ClusterBrokersOverviewVOList(List<Integer> pagedBrokerIdList,
+    private List<ClusterBrokersOverviewVO> convert2ClusterBrokersOverviewVOList(ClusterPhy clusterPhy,
+                                                                                List<Integer> pagedBrokerIdList,
                                                                                 List<Broker> brokerList,
                                                                                 List<BrokerMetrics> metricsList,
                                                                                 Topic groupTopic,
                                                                                 Topic transactionTopic,
-                                                                                KafkaController kafkaController) {
+                                                                                KafkaController kafkaController,
-        Map<Integer, BrokerMetrics> metricsMap = metricsList == null? new HashMap<>(): metricsList.stream().collect(Collectors.toMap(BrokerMetrics::getBrokerId, Function.identity()));
+                                                                                Map<Integer, Boolean> jmxConnectedMap) {
+        Map<Integer, BrokerMetrics> metricsMap = metricsList == null ? new HashMap<>() : metricsList.stream().collect(Collectors.toMap(BrokerMetrics::getBrokerId, Function.identity()));
 
-        Map<Integer, Broker> brokerMap = brokerList == null? new HashMap<>(): brokerList.stream().collect(Collectors.toMap(Broker::getBrokerId, Function.identity()));
+        Map<Integer, Broker> brokerMap = brokerList == null ? new HashMap<>() : brokerList.stream().collect(Collectors.toMap(Broker::getBrokerId, Function.identity()));
 
         List<ClusterBrokersOverviewVO> voList = new ArrayList<>(pagedBrokerIdList.size());
         for (Integer brokerId : pagedBrokerIdList) {
             Broker broker = brokerMap.get(brokerId);
             BrokerMetrics brokerMetrics = metricsMap.get(brokerId);
-
+            Boolean jmxConnected = jmxConnectedMap.get(brokerId);
-            voList.add(this.convert2ClusterBrokersOverviewVO(brokerId, broker, brokerMetrics, groupTopic, transactionTopic, kafkaController));
+            voList.add(this.convert2ClusterBrokersOverviewVO(brokerId, broker, brokerMetrics, groupTopic, transactionTopic, kafkaController, jmxConnected));
         }
+
+        //补充非zk模式的JMXPort信息
+        if (!clusterPhy.getRunState().equals(ClusterRunStateEnum.RUN_ZK.getRunState())) {
+            JmxConfig jmxConfig = ConvertUtil.str2ObjByJson(clusterPhy.getJmxProperties(), JmxConfig.class);
+            voList.forEach(elem -> elem.setJmxPort(jmxConfig.getJmxPort() == null ? -1 : jmxConfig.getJmxPort()));
+        }
+
         return voList;
     }
 
-    private ClusterBrokersOverviewVO convert2ClusterBrokersOverviewVO(Integer brokerId, Broker broker, BrokerMetrics brokerMetrics, Topic groupTopic, Topic transactionTopic, KafkaController kafkaController) {
+    private ClusterBrokersOverviewVO convert2ClusterBrokersOverviewVO(Integer brokerId, Broker broker, BrokerMetrics brokerMetrics, Topic groupTopic, Topic transactionTopic, KafkaController kafkaController, Boolean jmxConnected) {
         ClusterBrokersOverviewVO clusterBrokersOverviewVO = new ClusterBrokersOverviewVO();
         clusterBrokersOverviewVO.setBrokerId(brokerId);
         if (broker != null) {
@@ -203,6 +231,7 @@ public class ClusterBrokersManagerImpl implements ClusterBrokersManager {
         }
 
         clusterBrokersOverviewVO.setLatestMetrics(brokerMetrics);
+        clusterBrokersOverviewVO.setJmxConnected(jmxConnected);
         return clusterBrokersOverviewVO;
     }
 

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/cluster/impl/ClusterConnectorsManagerImpl.java

@@ -0,0 +1,152 @@
+package com.xiaojukeji.know.streaming.km.biz.cluster.impl;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.biz.cluster.ClusterConnectorsManager;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.ClusterConnectorsOverviewDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.ClusterConnectorDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.MetricDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.connect.MetricsConnectorsDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.ConnectCluster;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.ConnectWorker;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.WorkerConnector;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.connect.ConnectorMetrics;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
+import com.xiaojukeji.know.streaming.km.common.bean.po.connect.ConnectorPO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.connect.ConnectStateVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.connector.ClusterConnectorOverviewVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.metrics.line.MetricMultiLinesVO;
+import com.xiaojukeji.know.streaming.km.common.converter.ConnectConverter;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
+import com.xiaojukeji.know.streaming.km.common.utils.PaginationMetricsUtil;
+import com.xiaojukeji.know.streaming.km.common.utils.PaginationUtil;
+import com.xiaojukeji.know.streaming.km.core.service.connect.cluster.ConnectClusterService;
+import com.xiaojukeji.know.streaming.km.core.service.connect.connector.ConnectorMetricService;
+import com.xiaojukeji.know.streaming.km.core.service.connect.connector.ConnectorService;
+import com.xiaojukeji.know.streaming.km.core.service.connect.worker.WorkerConnectorService;
+import com.xiaojukeji.know.streaming.km.core.service.connect.worker.WorkerService;
+import org.apache.kafka.connect.runtime.AbstractStatus;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+
+@Service
+public class ClusterConnectorsManagerImpl implements ClusterConnectorsManager {
+    private static final ILog LOGGER = LogFactory.getLog(ClusterConnectorsManagerImpl.class);
+
+    @Autowired
+    private ConnectorService connectorService;
+
+    @Autowired
+    private ConnectClusterService connectClusterService;
+
+    @Autowired
+    private ConnectorMetricService connectorMetricService;
+
+    @Autowired
+    private WorkerService workerService;
+
+    @Autowired
+    private WorkerConnectorService workerConnectorService;
+
+    @Override
+    public PaginationResult<ClusterConnectorOverviewVO> getClusterConnectorsOverview(Long clusterPhyId, ClusterConnectorsOverviewDTO dto) {
+        List<ConnectCluster> clusterList = connectClusterService.listByKafkaCluster(clusterPhyId);
+
+        List<ConnectorPO> poList = connectorService.listByKafkaClusterIdFromDB(clusterPhyId);
+
+        // 查询实时指标
+        Result<List<ConnectorMetrics>> latestMetricsResult = connectorMetricService.getLatestMetricsFromES(
+                clusterPhyId,
+                poList.stream().map(elem -> new ClusterConnectorDTO(elem.getConnectClusterId(), elem.getConnectorName())).collect(Collectors.toList()),
+                dto.getLatestMetricNames()
+        );
+
+        if (latestMetricsResult.failed()) {
+            LOGGER.error("method=getClusterConnectorsOverview||clusterPhyId={}||result={}||errMsg=get latest metric failed", clusterPhyId, latestMetricsResult);
+            return PaginationResult.buildFailure(latestMetricsResult, dto);
+        }
+
+        // 转换成vo
+        List<ClusterConnectorOverviewVO> voList = ConnectConverter.convert2ClusterConnectorOverviewVOList(clusterList, poList,latestMetricsResult.getData());
+
+        // 请求分页信息
+        PaginationResult<ClusterConnectorOverviewVO> voPaginationResult = this.pagingConnectorInLocal(voList, dto);
+        if (voPaginationResult.failed()) {
+            LOGGER.error("method=getClusterConnectorsOverview||clusterPhyId={}||result={}||errMsg=pagination in local failed", clusterPhyId, voPaginationResult);
+
+            return PaginationResult.buildFailure(voPaginationResult, dto);
+        }
+
+        // 查询历史指标
+        Result<List<MetricMultiLinesVO>> lineMetricsResult = connectorMetricService.listConnectClusterMetricsFromES(
+                clusterPhyId,
+                this.buildMetricsConnectorsDTO(
+                        voPaginationResult.getData().getBizData().stream().map(elem -> new ClusterConnectorDTO(elem.getConnectClusterId(), elem.getConnectorName())).collect(Collectors.toList()),
+                        dto.getMetricLines()
+                )
+        );
+
+
+        return PaginationResult.buildSuc(
+                ConnectConverter.supplyData2ClusterConnectorOverviewVOList(
+                        voPaginationResult.getData().getBizData(),
+                        lineMetricsResult.getData()
+                ),
+                voPaginationResult
+        );
+    }
+
+    @Override
+    public ConnectStateVO getClusterConnectorsState(Long clusterPhyId) {
+        //获取Connect集群Id列表
+        List<ConnectCluster> connectClusterList = connectClusterService.listByKafkaCluster(clusterPhyId);
+        List<ConnectorPO> connectorPOList = connectorService.listByKafkaClusterIdFromDB(clusterPhyId);
+        List<WorkerConnector> workerConnectorList = workerConnectorService.listByKafkaClusterIdFromDB(clusterPhyId);
+        List<ConnectWorker> connectWorkerList = workerService.listByKafkaClusterIdFromDB(clusterPhyId);
+
+        return convert2ConnectStateVO(connectClusterList, connectorPOList, workerConnectorList, connectWorkerList);
+    }
+
+    /**************************************************** private method ****************************************************/
+
+    private MetricsConnectorsDTO buildMetricsConnectorsDTO(List<ClusterConnectorDTO> connectorDTOList, MetricDTO metricDTO) {
+        MetricsConnectorsDTO dto = ConvertUtil.obj2Obj(metricDTO, MetricsConnectorsDTO.class);
+        dto.setConnectorNameList(connectorDTOList == null? new ArrayList<>(): connectorDTOList);
+
+        return dto;
+    }
+
+    private ConnectStateVO convert2ConnectStateVO(List<ConnectCluster> connectClusterList, List<ConnectorPO> connectorPOList, List<WorkerConnector> workerConnectorList, List<ConnectWorker> connectWorkerList) {
+        ConnectStateVO connectStateVO = new ConnectStateVO();
+        connectStateVO.setConnectClusterCount(connectClusterList.size());
+        connectStateVO.setTotalConnectorCount(connectorPOList.size());
+        connectStateVO.setAliveConnectorCount(connectorPOList.stream().filter(elem -> elem.getState().equals(AbstractStatus.State.RUNNING.name())).collect(Collectors.toList()).size());
+        connectStateVO.setWorkerCount(connectWorkerList.size());
+        connectStateVO.setTotalTaskCount(workerConnectorList.size());
+        connectStateVO.setAliveTaskCount(workerConnectorList.stream().filter(elem -> elem.getState().equals(AbstractStatus.State.RUNNING.name())).collect(Collectors.toList()).size());
+        return connectStateVO;
+    }
+
+    private PaginationResult<ClusterConnectorOverviewVO> pagingConnectorInLocal(List<ClusterConnectorOverviewVO> connectorVOList, ClusterConnectorsOverviewDTO dto) {
+        //模糊匹配
+        connectorVOList = PaginationUtil.pageByFuzzyFilter(connectorVOList, dto.getSearchKeywords(), Arrays.asList("connectorName"));
+
+        //排序
+        if (!dto.getLatestMetricNames().isEmpty()) {
+            PaginationMetricsUtil.sortMetrics(connectorVOList, "latestMetrics", dto.getSortMetricNameList(), "connectorName", dto.getSortType());
+        } else {
+            PaginationUtil.pageBySort(connectorVOList, dto.getSortField(), dto.getSortType(), "connectorName", dto.getSortType());
+        }
+
+        //分页
+        return PaginationUtil.pageBySubData(connectorVOList, dto);
+    }
+
+}

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/cluster/impl/ClusterTopicsManagerImpl.java

@@ -14,10 +14,12 @@ import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.res.ClusterPhyTop
 import com.xiaojukeji.know.streaming.km.common.bean.vo.metrics.line.MetricMultiLinesVO;
 import com.xiaojukeji.know.streaming.km.common.constant.KafkaConstant;
 import com.xiaojukeji.know.streaming.km.common.converter.TopicVOConverter;
+import com.xiaojukeji.know.streaming.km.common.enums.ha.HaResTypeEnum;
 import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.PaginationMetricsUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.PaginationUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
+import com.xiaojukeji.know.streaming.km.core.service.ha.HaActiveStandbyRelationService;
 import com.xiaojukeji.know.streaming.km.core.service.topic.TopicMetricService;
 import com.xiaojukeji.know.streaming.km.core.service.topic.TopicService;
 import org.springframework.beans.factory.annotation.Autowired;
@@ -38,16 +40,22 @@ public class ClusterTopicsManagerImpl implements ClusterTopicsManager {
     @Autowired
     private TopicMetricService topicMetricService;
 
+    @Autowired
+    private HaActiveStandbyRelationService haActiveStandbyRelationService;
+
     @Override
     public PaginationResult<ClusterPhyTopicsOverviewVO> getClusterPhyTopicsOverview(Long clusterPhyId, ClusterTopicsOverviewDTO dto) {
         // 获取集群所有的Topic信息
         List<Topic> topicList = topicService.listTopicsFromDB(clusterPhyId);
 
         // 获取集群所有Topic的指标
-        Map<String, TopicMetrics> metricsMap = topicMetricService.getLatestMetricsFromCacheFirst(clusterPhyId);
+        Map<String, TopicMetrics> metricsMap = topicMetricService.getLatestMetricsFromCache(clusterPhyId);
+
+        // 获取HA信息
+        Set<String> haTopicNameSet = haActiveStandbyRelationService.listByClusterAndType(clusterPhyId, HaResTypeEnum.MIRROR_TOPIC).stream().map(elem -> elem.getResName()).collect(Collectors.toSet());
 
         // 转换成vo
-        List<ClusterPhyTopicsOverviewVO> voList = TopicVOConverter.convert2ClusterPhyTopicsOverviewVOList(topicList, metricsMap);
+        List<ClusterPhyTopicsOverviewVO> voList = TopicVOConverter.convert2ClusterPhyTopicsOverviewVOList(topicList, metricsMap, haTopicNameSet);
 
         // 请求分页信息
         PaginationResult<ClusterPhyTopicsOverviewVO> voPaginationResult = this.pagingTopicInLocal(voList, dto);

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/cluster/impl/ClusterZookeepersManagerImpl.java

@@ -0,0 +1,137 @@
+package com.xiaojukeji.know.streaming.km.biz.cluster.impl;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.biz.cluster.ClusterZookeepersManager;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.ClusterZookeepersOverviewDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.ZookeeperMetrics;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.ResultStatus;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.zookeeper.Znode;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.zookeeper.ZookeeperInfo;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.zookeeper.ClusterZookeepersOverviewVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.zookeeper.ClusterZookeepersStateVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.zookeeper.ZnodeVO;
+import com.xiaojukeji.know.streaming.km.common.constant.MsgConstant;
+import com.xiaojukeji.know.streaming.km.common.enums.zookeeper.ZKRoleEnum;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
+import com.xiaojukeji.know.streaming.km.common.utils.PaginationUtil;
+import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterPhyService;
+import com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.ZookeeperMetricVersionItems;
+import com.xiaojukeji.know.streaming.km.core.service.zookeeper.ZnodeService;
+import com.xiaojukeji.know.streaming.km.core.service.zookeeper.ZookeeperMetricService;
+import com.xiaojukeji.know.streaming.km.core.service.zookeeper.ZookeeperService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+import java.util.Arrays;
+import java.util.List;
+
+
+@Service
+public class ClusterZookeepersManagerImpl implements ClusterZookeepersManager {
+    private static final ILog LOGGER = LogFactory.getLog(ClusterZookeepersManagerImpl.class);
+
+    @Autowired
+    private ClusterPhyService clusterPhyService;
+
+    @Autowired
+    private ZookeeperService zookeeperService;
+
+    @Autowired
+    private ZookeeperMetricService zookeeperMetricService;
+
+    @Autowired
+    private ZnodeService znodeService;
+
+    @Override
+    public Result<ClusterZookeepersStateVO> getClusterPhyZookeepersState(Long clusterPhyId) {
+        ClusterPhy clusterPhy = clusterPhyService.getClusterByCluster(clusterPhyId);
+        if (clusterPhy == null) {
+            return Result.buildFromRSAndMsg(ResultStatus.CLUSTER_NOT_EXIST, MsgConstant.getClusterPhyNotExist(clusterPhyId));
+        }
+
+        List<ZookeeperInfo> infoList = zookeeperService.listFromDBByCluster(clusterPhyId);
+
+        ClusterZookeepersStateVO vo = new ClusterZookeepersStateVO();
+        vo.setTotalServerCount(infoList.size());
+        vo.setAliveFollowerCount(0);
+        vo.setTotalFollowerCount(0);
+        vo.setAliveObserverCount(0);
+        vo.setTotalObserverCount(0);
+        vo.setAliveServerCount(0);
+        for (ZookeeperInfo info: infoList) {
+            if (info.getRole().equals(ZKRoleEnum.LEADER.getRole())) {
+                vo.setLeaderNode(info.getHost());
+            }
+
+            if (info.getRole().equals(ZKRoleEnum.FOLLOWER.getRole())) {
+                vo.setTotalFollowerCount(vo.getTotalFollowerCount() + 1);
+                vo.setAliveFollowerCount(info.alive()? vo.getAliveFollowerCount() + 1: vo.getAliveFollowerCount());
+            }
+
+            if (info.getRole().equals(ZKRoleEnum.OBSERVER.getRole())) {
+                vo.setTotalObserverCount(vo.getTotalObserverCount() + 1);
+                vo.setAliveObserverCount(info.alive()? vo.getAliveObserverCount() + 1: vo.getAliveObserverCount());
+            }
+
+            if (info.alive()) {
+                vo.setAliveServerCount(vo.getAliveServerCount() + 1);
+            }
+        }
+
+        // 指标获取
+        Result<ZookeeperMetrics> metricsResult = zookeeperMetricService.batchCollectMetricsFromZookeeper(
+                clusterPhyId,
+                Arrays.asList(
+                        ZookeeperMetricVersionItems.ZOOKEEPER_METRIC_WATCH_COUNT,
+                        ZookeeperMetricVersionItems.ZOOKEEPER_METRIC_HEALTH_STATE,
+                        ZookeeperMetricVersionItems.ZOOKEEPER_METRIC_HEALTH_CHECK_PASSED,
+                        ZookeeperMetricVersionItems.ZOOKEEPER_METRIC_HEALTH_CHECK_TOTAL
+                )
+
+        );
+        if (metricsResult.failed()) {
+            LOGGER.error(
+                    "method=getClusterPhyZookeepersState||clusterPhyId={}||errMsg={}",
+                    clusterPhyId, metricsResult.getMessage()
+            );
+            return Result.buildSuc(vo);
+        }
+
+        ZookeeperMetrics metrics = metricsResult.getData();
+        vo.setWatchCount(ConvertUtil.float2Integer(metrics.getMetrics().get(ZookeeperMetricVersionItems.ZOOKEEPER_METRIC_WATCH_COUNT)));
+        vo.setHealthState(ConvertUtil.float2Integer(metrics.getMetrics().get(ZookeeperMetricVersionItems.ZOOKEEPER_METRIC_HEALTH_STATE)));
+        vo.setHealthCheckPassed(ConvertUtil.float2Integer(metrics.getMetrics().get(ZookeeperMetricVersionItems.ZOOKEEPER_METRIC_HEALTH_CHECK_PASSED)));
+        vo.setHealthCheckTotal(ConvertUtil.float2Integer(metrics.getMetrics().get(ZookeeperMetricVersionItems.ZOOKEEPER_METRIC_HEALTH_CHECK_TOTAL)));
+
+        return Result.buildSuc(vo);
+    }
+
+    @Override
+    public PaginationResult<ClusterZookeepersOverviewVO> getClusterPhyZookeepersOverview(Long clusterPhyId, ClusterZookeepersOverviewDTO dto) {
+        //获取集群zookeeper列表
+        List<ClusterZookeepersOverviewVO> clusterZookeepersOverviewVOList = ConvertUtil.list2List(zookeeperService.listFromDBByCluster(clusterPhyId), ClusterZookeepersOverviewVO.class);
+
+        //搜索
+        clusterZookeepersOverviewVOList = PaginationUtil.pageByFuzzyFilter(clusterZookeepersOverviewVOList, dto.getSearchKeywords(), Arrays.asList("host"));
+
+        //分页
+        PaginationResult<ClusterZookeepersOverviewVO> paginationResult = PaginationUtil.pageBySubData(clusterZookeepersOverviewVOList, dto);
+
+        return paginationResult;
+    }
+
+    @Override
+    public Result<ZnodeVO> getZnodeVO(Long clusterPhyId, String path) {
+        Result<Znode> result = znodeService.getZnode(clusterPhyId, path);
+        if (result.failed()) {
+            return Result.buildFromIgnoreData(result);
+        }
+        return Result.buildSuc(ConvertUtil.obj2ObjByJSON(result.getData(), ZnodeVO.class));
+    }
+
+    /**************************************************** private method ****************************************************/
+
+}

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/cluster/impl/MultiClusterPhyManagerImpl.java

@@ -5,32 +5,34 @@ import com.didiglobal.logi.log.LogFactory;
 import com.xiaojukeji.know.streaming.km.biz.cluster.MultiClusterPhyManager;
 import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.MetricDTO;
 import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.MetricsClusterPhyDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhysHealthState;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhysState;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
 import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.MultiClusterDashboardDTO;
-import com.xiaojukeji.know.streaming.km.common.bean.entity.kafkacontroller.KafkaController;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.ClusterMetrics;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.ClusterPhyBaseVO;
 import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.ClusterPhyDashboardVO;
 import com.xiaojukeji.know.streaming.km.common.bean.vo.metrics.line.MetricMultiLinesVO;
 import com.xiaojukeji.know.streaming.km.common.constant.Constant;
 import com.xiaojukeji.know.streaming.km.common.converter.ClusterVOConverter;
+import com.xiaojukeji.know.streaming.km.common.enums.health.HealthStateEnum;
 import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.PaginationUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.PaginationMetricsUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
 import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterMetricService;
 import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterPhyService;
-import com.xiaojukeji.know.streaming.km.core.service.kafkacontroller.KafkaControllerService;
+import com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.ClusterMetricVersionItems;
-import com.xiaojukeji.know.streaming.km.core.service.version.metrics.ClusterMetricVersionItems;
+import com.xiaojukeji.know.streaming.km.rebalance.algorithm.model.Resource;
+import com.xiaojukeji.know.streaming.km.rebalance.common.BalanceMetricConstant;
+import com.xiaojukeji.know.streaming.km.rebalance.common.bean.entity.ClusterBalanceItemState;
+import com.xiaojukeji.know.streaming.km.rebalance.core.service.ClusterBalanceService;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Service;
 
-import java.util.ArrayList;
+import java.util.*;
-import java.util.Arrays;
-import java.util.List;
-import java.util.Map;
 import java.util.stream.Collectors;
 
 @Service
@@ -44,33 +46,50 @@ public class MultiClusterPhyManagerImpl implements MultiClusterPhyManager {
     private ClusterMetricService clusterMetricService;
 
     @Autowired
-    private KafkaControllerService kafkaControllerService;
+    private ClusterBalanceService clusterBalanceService;
 
     @Override
     public ClusterPhysState getClusterPhysState() {
         List<ClusterPhy> clusterPhyList = clusterPhyService.listAllClusters();
+        ClusterPhysState physState = new ClusterPhysState(0, 0, 0, clusterPhyList.size());
 
-        Map<Long, KafkaController> controllerMap = kafkaControllerService.getKafkaControllersFromDB(
+        for (ClusterPhy clusterPhy : clusterPhyList) {
-                clusterPhyList.stream().map(elem -> elem.getId()).collect(Collectors.toList()),
+            ClusterMetrics metrics = clusterMetricService.getLatestMetricsFromCache(clusterPhy.getId());
-                false
+            Float state = metrics.getMetric(ClusterMetricVersionItems.CLUSTER_METRIC_HEALTH_STATE);
-        );
+            if (state == null) {
-
+                physState.setUnknownCount(physState.getUnknownCount() + 1);
-        // TODO 后续产品上，看是否需要增加一个未知的状态，否则新接入的集群，因为新接入的集群，数据存在延迟
+            } else if (state.intValue() == HealthStateEnum.DEAD.getDimension()) {
-        ClusterPhysState physState = new ClusterPhysState(0, 0, clusterPhyList.size());
-        for (ClusterPhy clusterPhy: clusterPhyList) {
-            KafkaController kafkaController = controllerMap.get(clusterPhy.getId());
-
-            if (kafkaController != null && !kafkaController.alive()) {
-                // 存在明确的信息表示controller挂了
-                physState.setDownCount(physState.getDownCount() + 1);
-            } else if ((System.currentTimeMillis() - clusterPhy.getCreateTime().getTime() >= 5 * 60 * 1000) && kafkaController == null) {
-                // 集群接入时间是在近5分钟内，同时kafkaController信息不存在，则设置为down
                 physState.setDownCount(physState.getDownCount() + 1);
             } else {
-                // 其他情况都设置为alive
                 physState.setLiveCount(physState.getLiveCount() + 1);
             }
         }
+        return physState;
+    }
+
+
+    @Override
+    public ClusterPhysHealthState getClusterPhysHealthState() {
+        List<ClusterPhy> clusterPhyList = clusterPhyService.listAllClusters();
+
+        ClusterPhysHealthState physState = new ClusterPhysHealthState(clusterPhyList.size());
+        for (ClusterPhy clusterPhy: clusterPhyList) {
+            ClusterMetrics metrics = clusterMetricService.getLatestMetricsFromCache(clusterPhy.getId());
+            Float state = metrics.getMetric(ClusterMetricVersionItems.CLUSTER_METRIC_HEALTH_STATE);
+            if (state == null) {
+                physState.setUnknownCount(physState.getUnknownCount() + 1);
+            } else if (state.intValue() == HealthStateEnum.GOOD.getDimension()) {
+                physState.setGoodCount(physState.getGoodCount() + 1);
+            } else if (state.intValue() == HealthStateEnum.MEDIUM.getDimension()) {
+                physState.setMediumCount(physState.getMediumCount() + 1);
+            } else if (state.intValue() == HealthStateEnum.POOR.getDimension()) {
+                physState.setPoorCount(physState.getPoorCount() + 1);
+            } else if (state.intValue() == HealthStateEnum.DEAD.getDimension()) {
+                physState.setDeadCount(physState.getDeadCount() + 1);
+            } else {
+                physState.setUnknownCount(physState.getUnknownCount() + 1);
+            }
+        }
 
         return physState;
     }
@@ -83,24 +102,6 @@ public class MultiClusterPhyManagerImpl implements MultiClusterPhyManager {
         // 转为vo格式，方便后续进行分页筛选等
         List<ClusterPhyDashboardVO> voList = ConvertUtil.list2List(clusterPhyList, ClusterPhyDashboardVO.class);
 
-        // TODO 后续产品上，看是否需要增加一个未知的状态，否则新接入的集群，因为新接入的集群，数据存在延迟
-        // 获取集群controller信息并补充到vo中,
-        Map<Long, KafkaController> controllerMap = kafkaControllerService.getKafkaControllersFromDB(clusterPhyList.stream().map(elem -> elem.getId()).collect(Collectors.toList()), false);
-        for (ClusterPhyDashboardVO vo: voList) {
-            KafkaController kafkaController = controllerMap.get(vo.getId());
-
-            if (kafkaController != null && !kafkaController.alive()) {
-                // 存在明确的信息表示controller挂了
-                vo.setAlive(Constant.DOWN);
-            } else if ((System.currentTimeMillis() - vo.getCreateTime().getTime() >= 5 * 60L * 1000L) && kafkaController == null) {
-                // 集群接入时间是在近5分钟内，同时kafkaController信息不存在，则设置为down
-                vo.setAlive(Constant.DOWN);
-            } else {
-                // 其他情况都设置为alive
-                vo.setAlive(Constant.ALIVE);
-            }
-        }
-
         // 本地分页过滤
         voList = this.getAndPagingDataInLocal(voList, dto);
 
@@ -125,6 +126,15 @@ public class MultiClusterPhyManagerImpl implements MultiClusterPhyManager {
         );
     }
 
+    @Override
+    public Result<List<ClusterPhyBaseVO>> getClusterPhysBasic() {
+        // 获取集群
+        List<ClusterPhy> clusterPhyList = clusterPhyService.listAllClusters();
+
+        // 转为vo格式，方便后续进行分页筛选等
+        return Result.buildSuc(ConvertUtil.list2List(clusterPhyList, ClusterPhyBaseVO.class));
+    }
+
 
     /**************************************************** private method ****************************************************/
 
@@ -149,12 +159,11 @@ public class MultiClusterPhyManagerImpl implements MultiClusterPhyManager {
         List<ClusterMetrics> metricsList = new ArrayList<>();
         for (ClusterPhyDashboardVO vo: voList) {
             ClusterMetrics clusterMetrics = clusterMetricService.getLatestMetricsFromCache(vo.getId());
-            if (!clusterMetrics.getMetrics().containsKey(ClusterMetricVersionItems.CLUSTER_METRIC_HEALTH_SCORE)) {
+            clusterMetrics.getMetrics().putIfAbsent(ClusterMetricVersionItems.CLUSTER_METRIC_HEALTH_STATE, (float) HealthStateEnum.UNKNOWN.getDimension());
-                Float alive = clusterMetrics.getMetrics().get(ClusterMetricVersionItems.CLUSTER_METRIC_ALIVE);
+
-                // 如果集群没有健康分，则设置一个默认的健康分数值
+            Result<ClusterMetrics> balanceMetricsResult = this.getClusterLoadReBalanceInfo(vo.getId());
-                clusterMetrics.putMetric(ClusterMetricVersionItems.CLUSTER_METRIC_HEALTH_SCORE,
+            if (balanceMetricsResult.hasData()) {
-                        (alive != null && alive <= 0)? 0.0f: Constant.DEFAULT_CLUSTER_HEALTH_SCORE.floatValue()
+                clusterMetrics.putMetric(balanceMetricsResult.getData().getMetrics());
-                );
             }
 
             metricsList.add(clusterMetrics);
@@ -178,4 +187,21 @@ public class MultiClusterPhyManagerImpl implements MultiClusterPhyManager {
         dto.setClusterPhyIds(clusterIdList);
         return dto;
     }
+
+    private Result<ClusterMetrics> getClusterLoadReBalanceInfo(Long clusterPhyId) {
+        Result<ClusterBalanceItemState> stateResult = clusterBalanceService.getItemStateFromCacheFirst(clusterPhyId);
+        if (stateResult.failed()) {
+            return Result.buildFromIgnoreData(stateResult);
+        }
+
+        ClusterBalanceItemState state = stateResult.getData();
+
+        ClusterMetrics metric = ClusterMetrics.initWithMetrics(clusterPhyId, BalanceMetricConstant.CLUSTER_METRIC_LOAD_RE_BALANCE_ENABLE, state.getEnable()? Constant.YES: Constant.NO);
+        metric.putMetric(BalanceMetricConstant.CLUSTER_METRIC_LOAD_RE_BALANCE_CPU, state.getResItemState(Resource.CPU).floatValue());
+        metric.putMetric(BalanceMetricConstant.CLUSTER_METRIC_LOAD_RE_BALANCE_NW_IN, state.getResItemState(Resource.NW_IN).floatValue());
+        metric.putMetric(BalanceMetricConstant.CLUSTER_METRIC_LOAD_RE_BALANCE_NW_OUT, state.getResItemState(Resource.NW_OUT).floatValue());
+        metric.putMetric(BalanceMetricConstant.CLUSTER_METRIC_LOAD_RE_BALANCE_DISK, state.getResItemState(Resource.DISK).floatValue());
+
+        return Result.buildSuc(metric);
+    }
 }

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/connect/connector/ConnectorManager.java

@@ -0,0 +1,16 @@
+package com.xiaojukeji.know.streaming.km.biz.connect.connector;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector.ConnectorCreateDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.connect.connector.ConnectorStateVO;
+
+import java.util.Properties;
+
+public interface ConnectorManager {
+    Result<Void> updateConnectorConfig(Long connectClusterId, String connectorName, Properties configs, String operator);
+
+    Result<Void> createConnector(ConnectorCreateDTO dto, String operator);
+    Result<Void> createConnector(ConnectorCreateDTO dto, String heartbeatName, String checkpointName, String operator);
+
+    Result<ConnectorStateVO> getConnectorStateVO(Long connectClusterId, String connectorName);
+}

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/connect/connector/WorkerConnectorManager.java

@@ -0,0 +1,16 @@
+package com.xiaojukeji.know.streaming.km.biz.connect.connector;
+
+
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.connect.task.KCTaskOverviewVO;
+
+import java.util.List;
+
+/**
+ * @author wyb
+ * @date 2022/11/14
+ */
+public interface WorkerConnectorManager {
+    Result<List<KCTaskOverviewVO>> getTaskOverview(Long connectClusterId, String connectorName);
+
+}

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/connect/connector/impl/ConnectorManagerImpl.java

@@ -0,0 +1,115 @@
+package com.xiaojukeji.know.streaming.km.biz.connect.connector.impl;
+
+import com.xiaojukeji.know.streaming.km.biz.connect.connector.ConnectorManager;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector.ConnectorCreateDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.WorkerConnector;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.config.ConnectConfigInfos;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.connector.KSConnector;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.connector.KSConnectorInfo;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.ResultStatus;
+import com.xiaojukeji.know.streaming.km.common.bean.po.connect.ConnectorPO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.connect.connector.ConnectorStateVO;
+import com.xiaojukeji.know.streaming.km.common.constant.connect.KafkaConnectConstant;
+import com.xiaojukeji.know.streaming.km.core.service.connect.connector.ConnectorService;
+import com.xiaojukeji.know.streaming.km.core.service.connect.plugin.PluginService;
+import com.xiaojukeji.know.streaming.km.core.service.connect.worker.WorkerConnectorService;
+import org.apache.kafka.connect.runtime.AbstractStatus;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import java.util.List;
+import java.util.Properties;
+import java.util.stream.Collectors;
+
+@Service
+public class ConnectorManagerImpl implements ConnectorManager {
+    @Autowired
+    private PluginService pluginService;
+
+    @Autowired
+    private ConnectorService connectorService;
+
+    @Autowired
+    private WorkerConnectorService workerConnectorService;
+
+    @Override
+    public Result<Void> updateConnectorConfig(Long connectClusterId, String connectorName, Properties configs, String operator) {
+        Result<ConnectConfigInfos> infosResult = pluginService.validateConfig(connectClusterId, configs);
+        if (infosResult.failed()) {
+            return Result.buildFromIgnoreData(infosResult);
+        }
+
+        if (infosResult.getData().getErrorCount() > 0) {
+            return Result.buildFromRSAndMsg(ResultStatus.PARAM_ILLEGAL, "Connector参数错误");
+        }
+
+        return connectorService.updateConnectorConfig(connectClusterId, connectorName, configs, operator);
+    }
+
+    @Override
+    public Result<Void> createConnector(ConnectorCreateDTO dto, String operator) {
+        dto.getConfigs().put(KafkaConnectConstant.MIRROR_MAKER_NAME_FIELD_NAME, dto.getConnectorName());
+
+        Result<KSConnectorInfo> createResult = connectorService.createConnector(dto.getConnectClusterId(), dto.getConnectorName(), dto.getConfigs(), operator);
+        if (createResult.failed()) {
+            return Result.buildFromIgnoreData(createResult);
+        }
+
+        Result<KSConnector> ksConnectorResult = connectorService.getAllConnectorInfoFromCluster(dto.getConnectClusterId(), dto.getConnectorName());
+        if (ksConnectorResult.failed()) {
+            return Result.buildFromRSAndMsg(ResultStatus.SUCCESS, "创建成功，但是获取元信息失败，页面元信息会存在1分钟延迟");
+        }
+
+        connectorService.addNewToDB(ksConnectorResult.getData());
+        return Result.buildSuc();
+    }
+
+    @Override
+    public Result<Void> createConnector(ConnectorCreateDTO dto, String heartbeatName, String checkpointName, String operator) {
+        dto.getConfigs().put(KafkaConnectConstant.MIRROR_MAKER_NAME_FIELD_NAME, dto.getConnectorName());
+
+        Result<KSConnectorInfo> createResult = connectorService.createConnector(dto.getConnectClusterId(), dto.getConnectorName(), dto.getConfigs(), operator);
+        if (createResult.failed()) {
+            return Result.buildFromIgnoreData(createResult);
+        }
+
+        Result<KSConnector> ksConnectorResult = connectorService.getAllConnectorInfoFromCluster(dto.getConnectClusterId(), dto.getConnectorName());
+        if (ksConnectorResult.failed()) {
+            return Result.buildFromRSAndMsg(ResultStatus.SUCCESS, "创建成功，但是获取元信息失败，页面元信息会存在1分钟延迟");
+        }
+
+        KSConnector connector = ksConnectorResult.getData();
+        connector.setCheckpointConnectorName(checkpointName);
+        connector.setHeartbeatConnectorName(heartbeatName);
+
+        connectorService.addNewToDB(connector);
+        return Result.buildSuc();
+    }
+
+
+    @Override
+    public Result<ConnectorStateVO> getConnectorStateVO(Long connectClusterId, String connectorName) {
+        ConnectorPO connectorPO = connectorService.getConnectorFromDB(connectClusterId, connectorName);
+
+        if (connectorPO == null) {
+            return Result.buildFailure(ResultStatus.NOT_EXIST);
+        }
+
+        List<WorkerConnector> workerConnectorList = workerConnectorService.listFromDB(connectClusterId).stream().filter(elem -> elem.getConnectorName().equals(connectorName)).collect(Collectors.toList());
+
+        return Result.buildSuc(convert2ConnectorOverviewVO(connectorPO, workerConnectorList));
+    }
+
+    private ConnectorStateVO convert2ConnectorOverviewVO(ConnectorPO connectorPO, List<WorkerConnector> workerConnectorList) {
+        ConnectorStateVO connectorStateVO = new ConnectorStateVO();
+        connectorStateVO.setConnectClusterId(connectorPO.getConnectClusterId());
+        connectorStateVO.setName(connectorPO.getConnectorName());
+        connectorStateVO.setType(connectorPO.getConnectorType());
+        connectorStateVO.setState(connectorPO.getState());
+        connectorStateVO.setTotalTaskCount(workerConnectorList.size());
+        connectorStateVO.setAliveTaskCount(workerConnectorList.stream().filter(elem -> elem.getState().equals(AbstractStatus.State.RUNNING.name())).collect(Collectors.toList()).size());
+        connectorStateVO.setTotalWorkerCount(workerConnectorList.stream().map(elem -> elem.getWorkerId()).collect(Collectors.toSet()).size());
+        return connectorStateVO;
+    }
+}

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/connect/connector/impl/WorkerConnectorManageImpl.java

@@ -0,0 +1,37 @@
+package com.xiaojukeji.know.streaming.km.biz.connect.connector.impl;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.biz.connect.connector.WorkerConnectorManager;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.ConnectCluster;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.WorkerConnector;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.connect.task.KCTaskOverviewVO;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
+import com.xiaojukeji.know.streaming.km.core.service.connect.worker.WorkerConnectorService;
+import com.xiaojukeji.know.streaming.km.persistence.connect.cache.LoadedConnectClusterCache;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import java.util.List;
+
+/**
+ * @author wyb
+ * @date 2022/11/14
+ */
+@Service
+public class WorkerConnectorManageImpl implements WorkerConnectorManager {
+
+    private static final ILog LOGGER = LogFactory.getLog(WorkerConnectorManageImpl.class);
+
+    @Autowired
+    private WorkerConnectorService workerConnectorService;
+
+    @Override
+    public Result<List<KCTaskOverviewVO>> getTaskOverview(Long connectClusterId, String connectorName) {
+        ConnectCluster connectCluster = LoadedConnectClusterCache.getByPhyId(connectClusterId);
+        List<WorkerConnector> workerConnectorList = workerConnectorService.getWorkerConnectorListFromCluster(connectCluster, connectorName);
+
+        return Result.buildSuc(ConvertUtil.list2List(workerConnectorList, KCTaskOverviewVO.class));
+    }
+}

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/connect/mm2/MirrorMakerManager.java

@@ -0,0 +1,43 @@
+package com.xiaojukeji.know.streaming.km.biz.connect.mm2;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.ClusterMirrorMakersOverviewDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.mm2.MirrorMakerCreateDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.mm2.ClusterMirrorMakerOverviewVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.mm2.MirrorMakerBaseStateVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.mm2.MirrorMakerStateVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.connect.plugin.ConnectConfigInfosVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.connect.task.KCTaskOverviewVO;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+/**
+ * @author wyb
+ * @date 2022/12/26
+ */
+public interface MirrorMakerManager {
+    Result<Void> createMirrorMaker(MirrorMakerCreateDTO dto, String operator);
+
+    Result<Void> deleteMirrorMaker(Long connectClusterId, String sourceConnectorName, String operator);
+
+    Result<Void> modifyMirrorMakerConfig(MirrorMakerCreateDTO dto, String operator);
+
+    Result<Void> restartMirrorMaker(Long connectClusterId, String sourceConnectorName, String operator);
+    Result<Void> stopMirrorMaker(Long connectClusterId, String sourceConnectorName, String operator);
+    Result<Void> resumeMirrorMaker(Long connectClusterId, String sourceConnectorName, String operator);
+
+    Result<MirrorMakerStateVO> getMirrorMakerStateVO(Long clusterPhyId);
+
+    PaginationResult<ClusterMirrorMakerOverviewVO> getClusterMirrorMakersOverview(Long clusterPhyId, ClusterMirrorMakersOverviewDTO dto);
+
+
+    Result<MirrorMakerBaseStateVO> getMirrorMakerState(Long connectId, String connectName);
+
+    Result<Map<String, List<KCTaskOverviewVO>>> getTaskOverview(Long connectClusterId, String connectorName);
+    Result<List<Properties>> getMM2Configs(Long connectClusterId, String connectorName);
+
+    Result<List<ConnectConfigInfosVO>> validateConnectors(MirrorMakerCreateDTO dto);
+}

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/connect/mm2/impl/MirrorMakerManagerImpl.java

@@ -0,0 +1,652 @@
+package com.xiaojukeji.know.streaming.km.biz.connect.mm2.impl;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.biz.connect.connector.ConnectorManager;
+import com.xiaojukeji.know.streaming.km.biz.connect.mm2.MirrorMakerManager;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.ClusterMirrorMakersOverviewDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.ClusterConnectorDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector.ConnectorCreateDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.mm2.MirrorMakerCreateDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.MetricDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.mm2.MetricsMirrorMakersDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.ConnectCluster;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.ConnectWorker;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.WorkerConnector;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.config.ConnectConfigInfos;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.connector.KSConnectorInfo;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.mm2.MirrorMakerMetrics;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.ResultStatus;
+import com.xiaojukeji.know.streaming.km.common.bean.po.connect.ConnectorPO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.mm2.ClusterMirrorMakerOverviewVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.mm2.MirrorMakerBaseStateVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.mm2.MirrorMakerStateVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.connect.plugin.ConnectConfigInfosVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.metrics.line.MetricLineVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.metrics.line.MetricMultiLinesVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.connect.task.KCTaskOverviewVO;
+import com.xiaojukeji.know.streaming.km.common.constant.MsgConstant;
+import com.xiaojukeji.know.streaming.km.common.utils.*;
+import com.xiaojukeji.know.streaming.km.common.constant.connect.KafkaConnectConstant;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
+import com.xiaojukeji.know.streaming.km.common.utils.MirrorMakerUtil;
+import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
+import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterPhyService;
+import com.xiaojukeji.know.streaming.km.core.service.connect.cluster.ConnectClusterService;
+import com.xiaojukeji.know.streaming.km.core.service.connect.connector.ConnectorService;
+import com.xiaojukeji.know.streaming.km.core.service.connect.mm2.MirrorMakerMetricService;
+import com.xiaojukeji.know.streaming.km.core.service.connect.plugin.PluginService;
+import com.xiaojukeji.know.streaming.km.core.service.connect.worker.WorkerConnectorService;
+import com.xiaojukeji.know.streaming.km.core.service.connect.worker.WorkerService;
+import com.xiaojukeji.know.streaming.km.core.utils.ApiCallThreadPoolService;
+import com.xiaojukeji.know.streaming.km.persistence.cache.LoadedClusterPhyCache;
+import org.apache.commons.lang.StringUtils;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import java.util.*;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.kafka.connect.runtime.AbstractStatus.State.RUNNING;
+import static com.xiaojukeji.know.streaming.km.common.constant.connect.KafkaConnectConstant.*;
+
+
+/**
+ * @author wyb
+ * @date 2022/12/26
+ */
+@Service
+public class MirrorMakerManagerImpl implements MirrorMakerManager {
+    private static final ILog LOGGER = LogFactory.getLog(MirrorMakerManagerImpl.class);
+
+    @Autowired
+    private ConnectorService connectorService;
+
+    @Autowired
+    private WorkerConnectorService workerConnectorService;
+
+    @Autowired
+    private WorkerService workerService;
+
+    @Autowired
+    private ConnectorManager connectorManager;
+
+    @Autowired
+    private ClusterPhyService clusterPhyService;
+
+    @Autowired
+    private MirrorMakerMetricService mirrorMakerMetricService;
+
+    @Autowired
+    private ConnectClusterService connectClusterService;
+
+    @Autowired
+    private PluginService pluginService;
+
+    @Override
+    public Result<Void> createMirrorMaker(MirrorMakerCreateDTO dto, String operator) {
+        // 检查基本参数
+        Result<Void> rv = this.checkCreateMirrorMakerParamAndUnifyData(dto);
+        if (rv.failed()) {
+            return rv;
+        }
+
+        // 创建MirrorSourceConnector
+        Result<Void> sourceConnectResult = connectorManager.createConnector(
+                dto,
+                dto.getCheckpointConnectorConfigs() != null? MirrorMakerUtil.genCheckpointName(dto.getConnectorName()): "",
+                dto.getHeartbeatConnectorConfigs() != null? MirrorMakerUtil.genHeartbeatName(dto.getConnectorName()): "",
+                operator
+        );
+        if (sourceConnectResult.failed()) {
+            // 创建失败, 直接返回
+            return Result.buildFromIgnoreData(sourceConnectResult);
+        }
+
+        // 创建 checkpoint 任务
+        Result<Void> checkpointResult = Result.buildSuc();
+        if (dto.getCheckpointConnectorConfigs() != null) {
+            checkpointResult = connectorManager.createConnector(
+                    new ConnectorCreateDTO(dto.getConnectClusterId(), MirrorMakerUtil.genCheckpointName(dto.getConnectorName()), dto.getCheckpointConnectorConfigs()),
+                    operator
+            );
+        }
+
+        // 创建 heartbeat 任务
+        Result<Void> heartbeatResult = Result.buildSuc();
+        if (dto.getHeartbeatConnectorConfigs() != null) {
+            heartbeatResult = connectorManager.createConnector(
+                    new ConnectorCreateDTO(dto.getConnectClusterId(), MirrorMakerUtil.genHeartbeatName(dto.getConnectorName()), dto.getHeartbeatConnectorConfigs()),
+                    operator
+            );
+        }
+
+        // 全都成功
+        if (checkpointResult.successful() && checkpointResult.successful()) {
+            return Result.buildSuc();
+        } else if (checkpointResult.failed() && checkpointResult.failed()) {
+            return Result.buildFromRSAndMsg(
+                    ResultStatus.KAFKA_CONNECTOR_OPERATE_FAILED,
+                    String.format("创建 checkpoint & heartbeat 失败.\n失败信息分别为：%s\n\n%s", checkpointResult.getMessage(), heartbeatResult.getMessage())
+            );
+        } else if (checkpointResult.failed()) {
+            return Result.buildFromRSAndMsg(
+                    ResultStatus.KAFKA_CONNECTOR_OPERATE_FAILED,
+                    String.format("创建 checkpoint 失败.\n失败信息分别为：%s", checkpointResult.getMessage())
+            );
+        } else{
+            return Result.buildFromRSAndMsg(
+                    ResultStatus.KAFKA_CONNECTOR_OPERATE_FAILED,
+                    String.format("创建 heartbeat 失败.\n失败信息分别为：%s", heartbeatResult.getMessage())
+            );
+        }
+    }
+
+    @Override
+    public Result<Void> deleteMirrorMaker(Long connectClusterId, String sourceConnectorName, String operator) {
+        ConnectorPO connectorPO = connectorService.getConnectorFromDB(connectClusterId, sourceConnectorName);
+        if (connectorPO == null) {
+            return Result.buildFromRSAndMsg(ResultStatus.NOT_EXIST, MsgConstant.getConnectorNotExist(connectClusterId, sourceConnectorName));
+        }
+
+        Result<Void> rv = Result.buildSuc();
+        if (!ValidateUtils.isBlank(connectorPO.getCheckpointConnectorName())) {
+            rv = connectorService.deleteConnector(connectClusterId, connectorPO.getCheckpointConnectorName(), operator);
+        }
+        if (rv.failed()) {
+            return rv;
+        }
+
+        if (!ValidateUtils.isBlank(connectorPO.getHeartbeatConnectorName())) {
+            rv = connectorService.deleteConnector(connectClusterId, connectorPO.getHeartbeatConnectorName(), operator);
+        }
+        if (rv.failed()) {
+            return rv;
+        }
+
+        return connectorService.deleteConnector(connectClusterId, sourceConnectorName, operator);
+    }
+
+    @Override
+    public Result<Void> modifyMirrorMakerConfig(MirrorMakerCreateDTO dto, String operator) {
+        ConnectorPO connectorPO = connectorService.getConnectorFromDB(dto.getConnectClusterId(), dto.getConnectorName());
+        if (connectorPO == null) {
+            return Result.buildFromRSAndMsg(ResultStatus.NOT_EXIST, MsgConstant.getConnectorNotExist(dto.getConnectClusterId(), dto.getConnectorName()));
+        }
+
+        Result<Void> rv = Result.buildSuc();
+        if (!ValidateUtils.isBlank(connectorPO.getCheckpointConnectorName()) && dto.getCheckpointConnectorConfigs() != null) {
+            rv = connectorService.updateConnectorConfig(dto.getConnectClusterId(), connectorPO.getCheckpointConnectorName(), dto.getCheckpointConnectorConfigs(), operator);
+        }
+        if (rv.failed()) {
+            return rv;
+        }
+
+        if (!ValidateUtils.isBlank(connectorPO.getHeartbeatConnectorName()) && dto.getHeartbeatConnectorConfigs() != null) {
+            rv = connectorService.updateConnectorConfig(dto.getConnectClusterId(), connectorPO.getHeartbeatConnectorName(), dto.getHeartbeatConnectorConfigs(), operator);
+        }
+        if (rv.failed()) {
+            return rv;
+        }
+
+        return connectorService.updateConnectorConfig(dto.getConnectClusterId(), dto.getConnectorName(), dto.getConfigs(), operator);
+    }
+
+    @Override
+    public Result<Void> restartMirrorMaker(Long connectClusterId, String sourceConnectorName, String operator) {
+        ConnectorPO connectorPO = connectorService.getConnectorFromDB(connectClusterId, sourceConnectorName);
+        if (connectorPO == null) {
+            return Result.buildFromRSAndMsg(ResultStatus.NOT_EXIST, MsgConstant.getConnectorNotExist(connectClusterId, sourceConnectorName));
+        }
+
+        Result<Void> rv = Result.buildSuc();
+        if (!ValidateUtils.isBlank(connectorPO.getCheckpointConnectorName())) {
+            rv = connectorService.restartConnector(connectClusterId, connectorPO.getCheckpointConnectorName(), operator);
+        }
+        if (rv.failed()) {
+            return rv;
+        }
+
+        if (!ValidateUtils.isBlank(connectorPO.getHeartbeatConnectorName())) {
+            rv = connectorService.restartConnector(connectClusterId, connectorPO.getHeartbeatConnectorName(), operator);
+        }
+        if (rv.failed()) {
+            return rv;
+        }
+
+        return connectorService.restartConnector(connectClusterId, sourceConnectorName, operator);
+    }
+
+    @Override
+    public Result<Void> stopMirrorMaker(Long connectClusterId, String sourceConnectorName, String operator) {
+        ConnectorPO connectorPO = connectorService.getConnectorFromDB(connectClusterId, sourceConnectorName);
+        if (connectorPO == null) {
+            return Result.buildFromRSAndMsg(ResultStatus.NOT_EXIST, MsgConstant.getConnectorNotExist(connectClusterId, sourceConnectorName));
+        }
+
+        Result<Void> rv = Result.buildSuc();
+        if (!ValidateUtils.isBlank(connectorPO.getCheckpointConnectorName())) {
+            rv = connectorService.stopConnector(connectClusterId, connectorPO.getCheckpointConnectorName(), operator);
+        }
+        if (rv.failed()) {
+            return rv;
+        }
+
+        if (!ValidateUtils.isBlank(connectorPO.getHeartbeatConnectorName())) {
+            rv = connectorService.stopConnector(connectClusterId, connectorPO.getHeartbeatConnectorName(), operator);
+        }
+        if (rv.failed()) {
+            return rv;
+        }
+
+        return connectorService.stopConnector(connectClusterId, sourceConnectorName, operator);
+    }
+
+    @Override
+    public Result<Void> resumeMirrorMaker(Long connectClusterId, String sourceConnectorName, String operator) {
+        ConnectorPO connectorPO = connectorService.getConnectorFromDB(connectClusterId, sourceConnectorName);
+        if (connectorPO == null) {
+            return Result.buildFromRSAndMsg(ResultStatus.NOT_EXIST, MsgConstant.getConnectorNotExist(connectClusterId, sourceConnectorName));
+        }
+
+        Result<Void> rv = Result.buildSuc();
+        if (!ValidateUtils.isBlank(connectorPO.getCheckpointConnectorName())) {
+            rv = connectorService.resumeConnector(connectClusterId, connectorPO.getCheckpointConnectorName(), operator);
+        }
+        if (rv.failed()) {
+            return rv;
+        }
+
+        if (!ValidateUtils.isBlank(connectorPO.getHeartbeatConnectorName())) {
+            rv = connectorService.resumeConnector(connectClusterId, connectorPO.getHeartbeatConnectorName(), operator);
+        }
+        if (rv.failed()) {
+            return rv;
+        }
+
+        return connectorService.resumeConnector(connectClusterId, sourceConnectorName, operator);
+    }
+
+    @Override
+    public Result<MirrorMakerStateVO> getMirrorMakerStateVO(Long clusterPhyId) {
+        List<ConnectorPO> connectorPOList = connectorService.listByKafkaClusterIdFromDB(clusterPhyId);
+        List<WorkerConnector> workerConnectorList = workerConnectorService.listByKafkaClusterIdFromDB(clusterPhyId);
+        List<ConnectWorker> workerList = workerService.listByKafkaClusterIdFromDB(clusterPhyId);
+
+        return Result.buildSuc(convert2MirrorMakerStateVO(connectorPOList, workerConnectorList, workerList));
+    }
+
+    @Override
+    public PaginationResult<ClusterMirrorMakerOverviewVO> getClusterMirrorMakersOverview(Long clusterPhyId, ClusterMirrorMakersOverviewDTO dto) {
+        List<ConnectorPO> mirrorMakerList = connectorService.listByKafkaClusterIdFromDB(clusterPhyId).stream().filter(elem -> elem.getConnectorClassName().equals(MIRROR_MAKER_SOURCE_CONNECTOR_TYPE)).collect(Collectors.toList());
+        List<ConnectCluster> connectClusterList = connectClusterService.listByKafkaCluster(clusterPhyId);
+
+
+        Result<List<MirrorMakerMetrics>> latestMetricsResult = mirrorMakerMetricService.getLatestMetricsFromES(clusterPhyId,
+                mirrorMakerList.stream().map(elem -> new Tuple<>(elem.getConnectClusterId(), elem.getConnectorName())).collect(Collectors.toList()),
+                dto.getLatestMetricNames());
+
+        if (latestMetricsResult.failed()) {
+            LOGGER.error("method=getClusterMirrorMakersOverview||clusterPhyId={}||result={}||errMsg=get latest metric failed", clusterPhyId, latestMetricsResult);
+            return PaginationResult.buildFailure(latestMetricsResult, dto);
+        }
+
+        List<ClusterMirrorMakerOverviewVO> mirrorMakerOverviewVOList = this.convert2ClusterMirrorMakerOverviewVO(mirrorMakerList, connectClusterList, latestMetricsResult.getData());
+
+        List<ClusterMirrorMakerOverviewVO> mirrorMakerVOList = this.completeClusterInfo(mirrorMakerOverviewVOList);
+
+        PaginationResult<ClusterMirrorMakerOverviewVO> voPaginationResult = this.pagingMirrorMakerInLocal(mirrorMakerVOList, dto);
+
+        if (voPaginationResult.failed()) {
+            LOGGER.error("method=ClusterMirrorMakerOverviewVO||clusterPhyId={}||result={}||errMsg=pagination in local failed", clusterPhyId, voPaginationResult);
+
+            return PaginationResult.buildFailure(voPaginationResult, dto);
+        }
+
+        // 查询历史指标
+        Result<List<MetricMultiLinesVO>> lineMetricsResult = mirrorMakerMetricService.listMirrorMakerClusterMetricsFromES(
+                clusterPhyId,
+                this.buildMetricsConnectorsDTO(
+                        voPaginationResult.getData().getBizData().stream().map(elem -> new ClusterConnectorDTO(elem.getConnectClusterId(), elem.getConnectorName())).collect(Collectors.toList()),
+                        dto.getMetricLines()
+                ));
+
+        return PaginationResult.buildSuc(
+                this.supplyData2ClusterMirrorMakerOverviewVOList(
+                        voPaginationResult.getData().getBizData(),
+                        lineMetricsResult.getData()
+                ),
+                voPaginationResult
+        );
+    }
+
+    @Override
+    public Result<MirrorMakerBaseStateVO> getMirrorMakerState(Long connectClusterId, String connectName) {
+        //mm2任务
+        ConnectorPO connectorPO = connectorService.getConnectorFromDB(connectClusterId, connectName);
+        if (connectorPO == null){
+            return Result.buildFrom(ResultStatus.NOT_EXIST);
+        }
+
+        List<WorkerConnector> workerConnectorList = workerConnectorService.listFromDB(connectClusterId).stream()
+                .filter(workerConnector -> workerConnector.getConnectorName().equals(connectorPO.getConnectorName())
+                        || (!StringUtils.isBlank(connectorPO.getCheckpointConnectorName()) && workerConnector.getConnectorName().equals(connectorPO.getCheckpointConnectorName()))
+                        || (!StringUtils.isBlank(connectorPO.getHeartbeatConnectorName()) && workerConnector.getConnectorName().equals(connectorPO.getHeartbeatConnectorName())))
+                        .collect(Collectors.toList());
+
+        MirrorMakerBaseStateVO mirrorMakerBaseStateVO = new MirrorMakerBaseStateVO();
+        mirrorMakerBaseStateVO.setTotalTaskCount(workerConnectorList.size());
+        mirrorMakerBaseStateVO.setAliveTaskCount(workerConnectorList.stream().filter(elem -> elem.getState().equals(RUNNING.name())).collect(Collectors.toList()).size());
+        mirrorMakerBaseStateVO.setWorkerCount(workerConnectorList.stream().collect(Collectors.groupingBy(WorkerConnector::getWorkerId)).size());
+        return Result.buildSuc(mirrorMakerBaseStateVO);
+    }
+
+    @Override
+    public Result<Map<String, List<KCTaskOverviewVO>>> getTaskOverview(Long connectClusterId, String connectorName) {
+        ConnectorPO connectorPO = connectorService.getConnectorFromDB(connectClusterId, connectorName);
+        if (connectorPO == null){
+            return Result.buildFrom(ResultStatus.NOT_EXIST);
+        }
+
+        Map<String, List<KCTaskOverviewVO>> listMap = new HashMap<>();
+        List<WorkerConnector> workerConnectorList = workerConnectorService.listFromDB(connectClusterId);
+        if (workerConnectorList.isEmpty()){
+            return Result.buildSuc(listMap);
+        }
+        workerConnectorList.forEach(workerConnector -> {
+            if (workerConnector.getConnectorName().equals(connectorPO.getConnectorName())){
+                listMap.putIfAbsent(KafkaConnectConstant.MIRROR_MAKER_SOURCE_CONNECTOR_TYPE, new ArrayList<>());
+                listMap.get(MIRROR_MAKER_SOURCE_CONNECTOR_TYPE).add(ConvertUtil.obj2Obj(workerConnector, KCTaskOverviewVO.class));
+            } else if (workerConnector.getConnectorName().equals(connectorPO.getCheckpointConnectorName())) {
+                listMap.putIfAbsent(KafkaConnectConstant.MIRROR_MAKER_HEARTBEAT_CONNECTOR_TYPE, new ArrayList<>());
+                listMap.get(MIRROR_MAKER_HEARTBEAT_CONNECTOR_TYPE).add(ConvertUtil.obj2Obj(workerConnector, KCTaskOverviewVO.class));
+            } else if (workerConnector.getConnectorName().equals(connectorPO.getHeartbeatConnectorName())) {
+                listMap.putIfAbsent(KafkaConnectConstant.MIRROR_MAKER_CHECKPOINT_CONNECTOR_TYPE, new ArrayList<>());
+                listMap.get(MIRROR_MAKER_CHECKPOINT_CONNECTOR_TYPE).add(ConvertUtil.obj2Obj(workerConnector, KCTaskOverviewVO.class));
+            }
+
+        });
+
+        return Result.buildSuc(listMap);
+    }
+
+    @Override
+    public Result<List<Properties>> getMM2Configs(Long connectClusterId, String connectorName) {
+        ConnectorPO connectorPO = connectorService.getConnectorFromDB(connectClusterId, connectorName);
+        if (connectorPO == null){
+            return Result.buildFrom(ResultStatus.NOT_EXIST);
+        }
+
+        List<Properties> propList = new ArrayList<>();
+
+        // source
+        Result<KSConnectorInfo> connectorResult = connectorService.getConnectorInfoFromCluster(connectClusterId, connectorPO.getConnectorName());
+        if (connectorResult.failed()) {
+            return Result.buildFromIgnoreData(connectorResult);
+        }
+
+        Properties props = new Properties();
+        props.putAll(connectorResult.getData().getConfig());
+        propList.add(props);
+
+        // checkpoint
+        if (!ValidateUtils.isBlank(connectorPO.getCheckpointConnectorName())) {
+            connectorResult = connectorService.getConnectorInfoFromCluster(connectClusterId, connectorPO.getCheckpointConnectorName());
+            if (connectorResult.failed()) {
+                return Result.buildFromIgnoreData(connectorResult);
+            }
+
+            props = new Properties();
+            props.putAll(connectorResult.getData().getConfig());
+            propList.add(props);
+        }
+
+
+        // heartbeat
+        if (!ValidateUtils.isBlank(connectorPO.getHeartbeatConnectorName())) {
+            connectorResult = connectorService.getConnectorInfoFromCluster(connectClusterId, connectorPO.getHeartbeatConnectorName());
+            if (connectorResult.failed()) {
+                return Result.buildFromIgnoreData(connectorResult);
+            }
+
+            props = new Properties();
+            props.putAll(connectorResult.getData().getConfig());
+            propList.add(props);
+        }
+
+        return Result.buildSuc(propList);
+    }
+
+    @Override
+    public Result<List<ConnectConfigInfosVO>> validateConnectors(MirrorMakerCreateDTO dto) {
+        List<ConnectConfigInfosVO> voList = new ArrayList<>();
+
+        Result<ConnectConfigInfos> infoResult = pluginService.validateConfig(dto.getConnectClusterId(), dto.getConfigs());
+        if (infoResult.failed()) {
+            return Result.buildFromIgnoreData(infoResult);
+        }
+
+        voList.add(ConvertUtil.obj2Obj(infoResult.getData(), ConnectConfigInfosVO.class));
+
+        if (dto.getHeartbeatConnectorConfigs() != null) {
+            infoResult = pluginService.validateConfig(dto.getConnectClusterId(), dto.getHeartbeatConnectorConfigs());
+            if (infoResult.failed()) {
+                return Result.buildFromIgnoreData(infoResult);
+            }
+
+            voList.add(ConvertUtil.obj2Obj(infoResult.getData(), ConnectConfigInfosVO.class));
+        }
+
+        if (dto.getCheckpointConnectorConfigs() != null) {
+            infoResult = pluginService.validateConfig(dto.getConnectClusterId(), dto.getCheckpointConnectorConfigs());
+            if (infoResult.failed()) {
+                return Result.buildFromIgnoreData(infoResult);
+            }
+
+            voList.add(ConvertUtil.obj2Obj(infoResult.getData(), ConnectConfigInfosVO.class));
+        }
+
+        return Result.buildSuc(voList);
+    }
+
+
+    /**************************************************** private method ****************************************************/
+
+    private MetricsMirrorMakersDTO buildMetricsConnectorsDTO(List<ClusterConnectorDTO> connectorDTOList, MetricDTO metricDTO) {
+        MetricsMirrorMakersDTO dto = ConvertUtil.obj2Obj(metricDTO, MetricsMirrorMakersDTO.class);
+        dto.setConnectorNameList(connectorDTOList == null? new ArrayList<>(): connectorDTOList);
+
+        return dto;
+    }
+
+    public Result<Void> checkCreateMirrorMakerParamAndUnifyData(MirrorMakerCreateDTO dto) {
+        ClusterPhy sourceClusterPhy = clusterPhyService.getClusterByCluster(dto.getSourceKafkaClusterId());
+        if (sourceClusterPhy == null) {
+            return Result.buildFromRSAndMsg(ResultStatus.CLUSTER_NOT_EXIST, MsgConstant.getClusterPhyNotExist(dto.getSourceKafkaClusterId()));
+        }
+
+        ConnectCluster connectCluster = connectClusterService.getById(dto.getConnectClusterId());
+        if (connectCluster == null) {
+            return Result.buildFromRSAndMsg(ResultStatus.CLUSTER_NOT_EXIST, MsgConstant.getConnectClusterNotExist(dto.getConnectClusterId()));
+        }
+
+        ClusterPhy targetClusterPhy = clusterPhyService.getClusterByCluster(connectCluster.getKafkaClusterPhyId());
+        if (targetClusterPhy == null) {
+            return Result.buildFromRSAndMsg(ResultStatus.CLUSTER_NOT_EXIST, MsgConstant.getClusterPhyNotExist(connectCluster.getKafkaClusterPhyId()));
+        }
+
+        if (!dto.getConfigs().containsKey(CONNECTOR_CLASS_FILED_NAME)) {
+            return Result.buildFromRSAndMsg(ResultStatus.PARAM_ILLEGAL, "SourceConnector缺少connector.class");
+        }
+
+        if (!MIRROR_MAKER_SOURCE_CONNECTOR_TYPE.equals(dto.getConfigs().getProperty(CONNECTOR_CLASS_FILED_NAME))) {
+            return Result.buildFromRSAndMsg(ResultStatus.PARAM_ILLEGAL, "SourceConnector的connector.class类型错误");
+        }
+
+        if (dto.getCheckpointConnectorConfigs() != null) {
+            if (!dto.getCheckpointConnectorConfigs().containsKey(CONNECTOR_CLASS_FILED_NAME)) {
+                return Result.buildFromRSAndMsg(ResultStatus.PARAM_ILLEGAL, "CheckpointConnector缺少connector.class");
+            }
+
+            if (!MIRROR_MAKER_CHECKPOINT_CONNECTOR_TYPE.equals(dto.getCheckpointConnectorConfigs().getProperty(CONNECTOR_CLASS_FILED_NAME))) {
+                return Result.buildFromRSAndMsg(ResultStatus.PARAM_ILLEGAL, "Checkpoint的connector.class类型错误");
+            }
+        }
+
+        if (dto.getHeartbeatConnectorConfigs() != null) {
+            if (!dto.getHeartbeatConnectorConfigs().containsKey(CONNECTOR_CLASS_FILED_NAME)) {
+                return Result.buildFromRSAndMsg(ResultStatus.PARAM_ILLEGAL, "HeartbeatConnector缺少connector.class");
+            }
+
+            if (!MIRROR_MAKER_HEARTBEAT_CONNECTOR_TYPE.equals(dto.getHeartbeatConnectorConfigs().getProperty(CONNECTOR_CLASS_FILED_NAME))) {
+                return Result.buildFromRSAndMsg(ResultStatus.PARAM_ILLEGAL, "Heartbeat的connector.class类型错误");
+            }
+        }
+
+        dto.unifyData(
+                sourceClusterPhy.getId(), sourceClusterPhy.getBootstrapServers(), ConvertUtil.str2ObjByJson(sourceClusterPhy.getClientProperties(), Properties.class),
+                targetClusterPhy.getId(), targetClusterPhy.getBootstrapServers(), ConvertUtil.str2ObjByJson(targetClusterPhy.getClientProperties(), Properties.class)
+        );
+
+        return Result.buildSuc();
+    }
+
+    private MirrorMakerStateVO convert2MirrorMakerStateVO(List<ConnectorPO> connectorPOList,List<WorkerConnector> workerConnectorList,List<ConnectWorker> workerList){
+        MirrorMakerStateVO mirrorMakerStateVO = new MirrorMakerStateVO();
+
+        List<ConnectorPO> sourceSet = connectorPOList.stream().filter(elem -> elem.getConnectorClassName().equals(MIRROR_MAKER_SOURCE_CONNECTOR_TYPE)).collect(Collectors.toList());
+        mirrorMakerStateVO.setMirrorMakerCount(sourceSet.size());
+
+        Set<Long> connectClusterIdSet = sourceSet.stream().map(ConnectorPO::getConnectClusterId).collect(Collectors.toSet());
+        mirrorMakerStateVO.setWorkerCount(workerList.stream().filter(elem -> connectClusterIdSet.contains(elem.getConnectClusterId())).collect(Collectors.toList()).size());
+
+        List<ConnectorPO> mirrorMakerConnectorList = new ArrayList<>();
+        mirrorMakerConnectorList.addAll(sourceSet);
+        mirrorMakerConnectorList.addAll(connectorPOList.stream().filter(elem -> elem.getConnectorClassName().equals(MIRROR_MAKER_CHECKPOINT_CONNECTOR_TYPE)).collect(Collectors.toList()));
+        mirrorMakerConnectorList.addAll(connectorPOList.stream().filter(elem -> elem.getConnectorClassName().equals(MIRROR_MAKER_HEARTBEAT_CONNECTOR_TYPE)).collect(Collectors.toList()));
+        mirrorMakerStateVO.setTotalConnectorCount(mirrorMakerConnectorList.size());
+        mirrorMakerStateVO.setAliveConnectorCount(mirrorMakerConnectorList.stream().filter(elem -> elem.getState().equals(RUNNING.name())).collect(Collectors.toList()).size());
+
+        Set<String> connectorNameSet = mirrorMakerConnectorList.stream().map(elem -> elem.getConnectorName()).collect(Collectors.toSet());
+        List<WorkerConnector> taskList = workerConnectorList.stream().filter(elem -> connectorNameSet.contains(elem.getConnectorName())).collect(Collectors.toList());
+        mirrorMakerStateVO.setTotalTaskCount(taskList.size());
+        mirrorMakerStateVO.setAliveTaskCount(taskList.stream().filter(elem -> elem.getState().equals(RUNNING.name())).collect(Collectors.toList()).size());
+
+        return mirrorMakerStateVO;
+    }
+
+    private List<ClusterMirrorMakerOverviewVO> convert2ClusterMirrorMakerOverviewVO(List<ConnectorPO> mirrorMakerList, List<ConnectCluster> connectClusterList, List<MirrorMakerMetrics> latestMetric) {
+        List<ClusterMirrorMakerOverviewVO> clusterMirrorMakerOverviewVOList = new ArrayList<>();
+        Map<String, MirrorMakerMetrics> metricsMap = latestMetric.stream().collect(Collectors.toMap(elem -> elem.getConnectClusterId() + "@" + elem.getConnectorName(), Function.identity()));
+        Map<Long, ConnectCluster> connectClusterMap = connectClusterList.stream().collect(Collectors.toMap(elem -> elem.getId(), Function.identity()));
+
+        for (ConnectorPO mirrorMaker : mirrorMakerList) {
+            ClusterMirrorMakerOverviewVO clusterMirrorMakerOverviewVO = new ClusterMirrorMakerOverviewVO();
+            clusterMirrorMakerOverviewVO.setConnectClusterId(mirrorMaker.getConnectClusterId());
+            clusterMirrorMakerOverviewVO.setConnectClusterName(connectClusterMap.get(mirrorMaker.getConnectClusterId()).getName());
+            clusterMirrorMakerOverviewVO.setConnectorName(mirrorMaker.getConnectorName());
+            clusterMirrorMakerOverviewVO.setState(mirrorMaker.getState());
+            clusterMirrorMakerOverviewVO.setCheckpointConnector(mirrorMaker.getCheckpointConnectorName());
+            clusterMirrorMakerOverviewVO.setTaskCount(mirrorMaker.getTaskCount());
+            clusterMirrorMakerOverviewVO.setHeartbeatConnector(mirrorMaker.getHeartbeatConnectorName());
+            clusterMirrorMakerOverviewVO.setLatestMetrics(metricsMap.getOrDefault(mirrorMaker.getConnectClusterId() + "@" + mirrorMaker.getConnectorName(), new MirrorMakerMetrics(mirrorMaker.getConnectClusterId(), mirrorMaker.getConnectorName())));
+            clusterMirrorMakerOverviewVOList.add(clusterMirrorMakerOverviewVO);
+        }
+        return clusterMirrorMakerOverviewVOList;
+    }
+
+    PaginationResult<ClusterMirrorMakerOverviewVO> pagingMirrorMakerInLocal(List<ClusterMirrorMakerOverviewVO> mirrorMakerOverviewVOList, ClusterMirrorMakersOverviewDTO dto) {
+        List<ClusterMirrorMakerOverviewVO> mirrorMakerVOList = PaginationUtil.pageByFuzzyFilter(mirrorMakerOverviewVOList, dto.getSearchKeywords(), Arrays.asList("connectorName"));
+
+        //排序
+        if (!dto.getLatestMetricNames().isEmpty()) {
+            PaginationMetricsUtil.sortMetrics(mirrorMakerVOList, "latestMetrics", dto.getSortMetricNameList(), "connectorName", dto.getSortType());
+        } else {
+            PaginationUtil.pageBySort(mirrorMakerVOList, dto.getSortField(), dto.getSortType(), "connectorName", dto.getSortType());
+        }
+
+        //分页
+        return PaginationUtil.pageBySubData(mirrorMakerVOList, dto);
+    }
+
+    public static List<ClusterMirrorMakerOverviewVO> supplyData2ClusterMirrorMakerOverviewVOList(List<ClusterMirrorMakerOverviewVO> voList,
+                                                                                                 List<MetricMultiLinesVO> metricLineVOList) {
+        Map<String, List<MetricLineVO>> metricLineMap = new HashMap<>();
+        if (metricLineVOList != null) {
+            for (MetricMultiLinesVO metricMultiLinesVO : metricLineVOList) {
+                metricMultiLinesVO.getMetricLines()
+                        .forEach(metricLineVO -> {
+                            String key = metricLineVO.getName();
+                            List<MetricLineVO> metricLineVOS = metricLineMap.getOrDefault(key, new ArrayList<>());
+                            metricLineVOS.add(metricLineVO);
+                            metricLineMap.put(key, metricLineVOS);
+                        });
+            }
+        }
+
+        voList.forEach(elem -> {
+            elem.setMetricLines(metricLineMap.get(elem.getConnectClusterId() + "#" + elem.getConnectorName()));
+        });
+
+        return voList;
+    }
+
+    private List<ClusterMirrorMakerOverviewVO> completeClusterInfo(List<ClusterMirrorMakerOverviewVO> mirrorMakerVOList) {
+
+        Map<String, KSConnectorInfo> connectorInfoMap = new HashMap<>();
+
+        for (ClusterMirrorMakerOverviewVO mirrorMakerVO : mirrorMakerVOList) {
+            ApiCallThreadPoolService.runnableTask(String.format("method=completeClusterInfo||connectClusterId=%d||connectorName=%s||getMirrorMakerInfo", mirrorMakerVO.getConnectClusterId(), mirrorMakerVO.getConnectorName()),
+                    3000
+                    , () -> {
+                        Result<KSConnectorInfo> connectorInfoRet = connectorService.getConnectorInfoFromCluster(mirrorMakerVO.getConnectClusterId(), mirrorMakerVO.getConnectorName());
+                        if (connectorInfoRet.hasData()) {
+                            connectorInfoMap.put(mirrorMakerVO.getConnectClusterId() + mirrorMakerVO.getConnectorName(), connectorInfoRet.getData());
+                        }
+
+                        return connectorInfoRet.getData();
+                    });
+        }
+
+        ApiCallThreadPoolService.waitResult(1000);
+
+        List<ClusterMirrorMakerOverviewVO> newMirrorMakerVOList = new ArrayList<>();
+        for (ClusterMirrorMakerOverviewVO mirrorMakerVO : mirrorMakerVOList) {
+            KSConnectorInfo connectorInfo = connectorInfoMap.get(mirrorMakerVO.getConnectClusterId() + mirrorMakerVO.getConnectorName());
+            if (connectorInfo == null) {
+                continue;
+            }
+
+            String sourceClusterAlias = connectorInfo.getConfig().get(MIRROR_MAKER_SOURCE_CLUSTER_ALIAS_FIELD_NAME);
+            String targetClusterAlias = connectorInfo.getConfig().get(MIRROR_MAKER_TARGET_CLUSTER_ALIAS_FIELD_NAME);
+            //先默认设置为集群别名
+            mirrorMakerVO.setSourceKafkaClusterName(sourceClusterAlias);
+            mirrorMakerVO.setDestKafkaClusterName(targetClusterAlias);
+
+            if (!ValidateUtils.isBlank(sourceClusterAlias) && CommonUtils.isNumeric(sourceClusterAlias)) {
+                ClusterPhy clusterPhy = LoadedClusterPhyCache.getByPhyId(Long.valueOf(sourceClusterAlias));
+                if (clusterPhy != null) {
+                    mirrorMakerVO.setSourceKafkaClusterId(clusterPhy.getId());
+                    mirrorMakerVO.setSourceKafkaClusterName(clusterPhy.getName());
+                }
+            }
+
+            if (!ValidateUtils.isBlank(targetClusterAlias) && CommonUtils.isNumeric(targetClusterAlias)) {
+                ClusterPhy clusterPhy = LoadedClusterPhyCache.getByPhyId(Long.valueOf(targetClusterAlias));
+                if (clusterPhy != null) {
+                    mirrorMakerVO.setDestKafkaClusterId(clusterPhy.getId());
+                    mirrorMakerVO.setDestKafkaClusterName(clusterPhy.getName());
+                }
+            }
+
+            newMirrorMakerVOList.add(mirrorMakerVO);
+
+        }
+
+        return newMirrorMakerVOList;
+    }
+}

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/group/GroupManager.java

@@ -1,11 +1,14 @@
 package com.xiaojukeji.know.streaming.km.biz.group;
 
+import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.ClusterGroupSummaryDTO;
 import com.xiaojukeji.know.streaming.km.common.bean.dto.group.GroupOffsetResetDTO;
 import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationBaseDTO;
 import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationSortDTO;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.topic.TopicPartitionKS;
+import com.xiaojukeji.know.streaming.km.common.bean.po.group.GroupMemberPO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.group.GroupOverviewVO;
 import com.xiaojukeji.know.streaming.km.common.bean.vo.group.GroupTopicConsumedDetailVO;
 import com.xiaojukeji.know.streaming.km.common.bean.vo.group.GroupTopicOverviewVO;
 import com.xiaojukeji.know.streaming.km.common.exception.AdminOperateException;
@@ -22,6 +25,10 @@ public interface GroupManager {
                                                               String searchGroupKeyword,
                                                               PaginationBaseDTO dto);
 
+    PaginationResult<GroupTopicOverviewVO> pagingGroupTopicMembers(Long clusterPhyId, String groupName, PaginationBaseDTO dto);
+
+    PaginationResult<GroupOverviewVO> pagingClusterGroupsOverview(Long clusterPhyId, ClusterGroupSummaryDTO dto);
+
     PaginationResult<GroupTopicConsumedDetailVO> pagingGroupTopicConsumedMetrics(Long clusterPhyId,
                                                                                  String topicName,
                                                                                  String groupName,
@@ -31,4 +38,6 @@ public interface GroupManager {
     Result<Set<TopicPartitionKS>> listClusterPhyGroupPartitions(Long clusterPhyId, String groupName, Long startTime, Long endTime);
 
     Result<Void> resetGroupOffsets(GroupOffsetResetDTO dto, String operator) throws Exception;
+
+    List<GroupTopicOverviewVO> getGroupTopicOverviewVOList(Long clusterPhyId, List<GroupMemberPO> groupMemberPOList);
 }

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/group/impl/GroupManagerImpl.java

@@ -3,23 +3,35 @@ package com.xiaojukeji.know.streaming.km.biz.group.impl;
 import com.didiglobal.logi.log.ILog;
 import com.didiglobal.logi.log.LogFactory;
 import com.xiaojukeji.know.streaming.km.biz.group.GroupManager;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.ClusterGroupSummaryDTO;
 import com.xiaojukeji.know.streaming.km.common.bean.dto.group.GroupOffsetResetDTO;
 import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationBaseDTO;
 import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationSortDTO;
 import com.xiaojukeji.know.streaming.km.common.bean.dto.partition.PartitionOffsetDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.group.Group;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.group.GroupTopic;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.group.GroupTopicMember;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.kafka.KSGroupDescription;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.kafka.KSMemberConsumerAssignment;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.kafka.KSMemberDescription;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.GroupMetrics;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.offset.KSOffsetSpec;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.topic.Topic;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.topic.TopicPartitionKS;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.result.ResultStatus;
 import com.xiaojukeji.know.streaming.km.common.bean.po.group.GroupMemberPO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.group.GroupOverviewVO;
 import com.xiaojukeji.know.streaming.km.common.bean.vo.group.GroupTopicConsumedDetailVO;
 import com.xiaojukeji.know.streaming.km.common.bean.vo.group.GroupTopicOverviewVO;
 import com.xiaojukeji.know.streaming.km.common.constant.MsgConstant;
+import com.xiaojukeji.know.streaming.km.common.constant.PaginationConstant;
+import com.xiaojukeji.know.streaming.km.common.converter.GroupConverter;
 import com.xiaojukeji.know.streaming.km.common.enums.AggTypeEnum;
 import com.xiaojukeji.know.streaming.km.common.enums.OffsetTypeEnum;
+import com.xiaojukeji.know.streaming.km.common.enums.SortTypeEnum;
 import com.xiaojukeji.know.streaming.km.common.enums.group.GroupStateEnum;
 import com.xiaojukeji.know.streaming.km.common.exception.AdminOperateException;
 import com.xiaojukeji.know.streaming.km.common.exception.NotExistException;
@@ -27,15 +39,13 @@ import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.PaginationMetricsUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.PaginationUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
+import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterPhyService;
 import com.xiaojukeji.know.streaming.km.core.service.group.GroupMetricService;
 import com.xiaojukeji.know.streaming.km.core.service.group.GroupService;
 import com.xiaojukeji.know.streaming.km.core.service.partition.PartitionService;
 import com.xiaojukeji.know.streaming.km.core.service.topic.TopicService;
-import com.xiaojukeji.know.streaming.km.core.service.version.metrics.GroupMetricVersionItems;
+import com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.GroupMetricVersionItems;
 import com.xiaojukeji.know.streaming.km.persistence.es.dao.GroupMetricESDAO;
-import org.apache.kafka.clients.admin.ConsumerGroupDescription;
-import org.apache.kafka.clients.admin.MemberDescription;
-import org.apache.kafka.clients.admin.OffsetSpec;
 import org.apache.kafka.common.ConsumerGroupState;
 import org.apache.kafka.common.TopicPartition;
 import org.springframework.beans.factory.annotation.Autowired;
@@ -44,6 +54,8 @@ import org.springframework.stereotype.Component;
 import java.util.*;
 import java.util.stream.Collectors;
 
+import static com.xiaojukeji.know.streaming.km.common.enums.group.GroupTypeEnum.CONNECT_CLUSTER_PROTOCOL_TYPE;
+
 @Component
 public class GroupManagerImpl implements GroupManager {
     private static final ILog log = LogFactory.getLog(GroupManagerImpl.class);
@@ -63,6 +75,9 @@ public class GroupManagerImpl implements GroupManager {
     @Autowired
     private GroupMetricESDAO groupMetricESDAO;
 
+    @Autowired
+    private ClusterPhyService clusterPhyService;
+
     @Override
     public PaginationResult<GroupTopicOverviewVO> pagingGroupMembers(Long clusterPhyId,
                                                                      String topicName,
@@ -71,30 +86,60 @@ public class GroupManagerImpl implements GroupManager {
                                                                      String searchGroupKeyword,
                                                                      PaginationBaseDTO dto) {
         PaginationResult<GroupMemberPO> paginationResult = groupService.pagingGroupMembers(clusterPhyId, topicName, groupName, searchTopicKeyword, searchGroupKeyword, dto);
-        if (paginationResult.failed()) {
-            return PaginationResult.buildFailure(paginationResult, dto);
-        }
 
         if (!paginationResult.hasData()) {
             return PaginationResult.buildSuc(new ArrayList<>(), paginationResult);
         }
 
-        // 获取指标
+        List<GroupTopicOverviewVO> groupTopicVOList = this.getGroupTopicOverviewVOList(clusterPhyId, paginationResult.getData().getBizData());
-        Result<List<GroupMetrics>> metricsListResult = groupMetricService.listLatestMetricsAggByGroupTopicFromES(
+
-                clusterPhyId,
+        return PaginationResult.buildSuc(groupTopicVOList, paginationResult);
-                paginationResult.getData().getBizData().stream().map(elem -> new GroupTopic(elem.getGroupName(), elem.getTopicName())).collect(Collectors.toList()),
+    }
-                Arrays.asList(GroupMetricVersionItems.GROUP_METRIC_LAG),
+
-                AggTypeEnum.MAX
+    @Override
-        );
+    public PaginationResult<GroupTopicOverviewVO> pagingGroupTopicMembers(Long clusterPhyId, String groupName, PaginationBaseDTO dto) {
-        if (metricsListResult.failed()) {
+        Group group = groupService.getGroupFromDB(clusterPhyId, groupName);
-            // 如果查询失败，则输出错误信息，但是依旧进行已有数据的返回
+
-            log.error("method=pagingGroupMembers||clusterPhyId={}||topicName={}||groupName={}||result={}||errMsg=search es failed", clusterPhyId, topicName, groupName, metricsListResult);
+        //没有topicMember则直接返回
+        if (group == null || ValidateUtils.isEmptyList(group.getTopicMembers())) {
+            return PaginationResult.buildSuc(dto);
         }
 
-        return PaginationResult.buildSuc(
+        //排序
-                this.convert2GroupTopicOverviewVOList(paginationResult.getData().getBizData(), metricsListResult.getData()),
+        List<GroupTopicMember> groupTopicMembers = PaginationUtil.pageBySort(group.getTopicMembers(), PaginationConstant.DEFAULT_GROUP_TOPIC_SORTED_FIELD, SortTypeEnum.DESC.getSortType());
-                paginationResult
+
-        );
+        //分页
+        PaginationResult<GroupTopicMember> paginationResult = PaginationUtil.pageBySubData(groupTopicMembers, dto);
+
+        List<GroupMemberPO> groupMemberPOList = paginationResult.getData().getBizData().stream().map(elem -> new GroupMemberPO(clusterPhyId, elem.getTopicName(), groupName, group.getState().getState(), elem.getMemberCount())).collect(Collectors.toList());
+
+        return PaginationResult.buildSuc(this.getGroupTopicOverviewVOList(clusterPhyId, groupMemberPOList), paginationResult);
+    }
+
+    @Override
+    public PaginationResult<GroupOverviewVO> pagingClusterGroupsOverview(Long clusterPhyId, ClusterGroupSummaryDTO dto) {
+        List<Group> groupList = groupService.listClusterGroups(clusterPhyId);
+
+        // 类型转化
+        List<GroupOverviewVO> voList = groupList.stream().map(elem -> GroupConverter.convert2GroupOverviewVO(elem)).collect(Collectors.toList());
+
+        // 搜索groupName
+        voList = PaginationUtil.pageByFuzzyFilter(voList, dto.getSearchGroupName(), Arrays.asList("name"));
+
+        //搜索topic
+        if (!ValidateUtils.isBlank(dto.getSearchTopicName())) {
+            voList = voList.stream().filter(elem -> {
+                for (String topicName : elem.getTopicNameList()) {
+                    if (topicName.contains(dto.getSearchTopicName())) {
+                        return true;
+                    }
+                }
+                return false;
+            }).collect(Collectors.toList());
+        }
+
+        // 分页 后 返回
+        return PaginationUtil.pageBySubData(voList, dto);
     }
 
     @Override
@@ -103,8 +148,13 @@ public class GroupManagerImpl implements GroupManager {
                                                                                         String groupName,
                                                                                         List<String> latestMetricNames,
                                                                                         PaginationSortDTO dto) throws NotExistException, AdminOperateException {
+        ClusterPhy clusterPhy = clusterPhyService.getClusterByCluster(clusterPhyId);
+        if (clusterPhy == null) {
+            return PaginationResult.buildFailure(MsgConstant.getClusterPhyNotExist(clusterPhyId), dto);
+        }
+
         // 获取消费组消费的TopicPartition列表
-        Map<TopicPartition, Long> consumedOffsetMap = groupService.getGroupOffset(clusterPhyId, groupName);
+        Map<TopicPartition, Long> consumedOffsetMap = groupService.getGroupOffsetFromKafka(clusterPhyId, groupName);
         List<Integer> partitionList = consumedOffsetMap.keySet()
                 .stream()
                 .filter(elem -> elem.topic().equals(topicName))
@@ -113,13 +163,18 @@ public class GroupManagerImpl implements GroupManager {
         Collections.sort(partitionList);
 
         // 获取消费组当前运行信息
-        ConsumerGroupDescription groupDescription = groupService.getGroupDescription(clusterPhyId, groupName);
+        KSGroupDescription groupDescription = groupService.getGroupDescriptionFromKafka(clusterPhy, groupName);
 
         // 转换存储格式
-        Map<TopicPartition, MemberDescription> tpMemberMap = new HashMap<>();
+        Map<TopicPartition, KSMemberDescription> tpMemberMap = new HashMap<>();
-        for (MemberDescription description: groupDescription.members()) {
+
-            for (TopicPartition tp: description.assignment().topicPartitions()) {
+        //如果不是connect集群
-                tpMemberMap.put(tp, description);
+        if (!groupDescription.protocolType().equals(CONNECT_CLUSTER_PROTOCOL_TYPE)) {
+            for (KSMemberDescription description : groupDescription.members()) {
+                KSMemberConsumerAssignment assignment = (KSMemberConsumerAssignment) description.assignment();
+                for (TopicPartition tp : assignment.topicPartitions()) {
+                    tpMemberMap.put(tp, description);
+                }
             }
         }
 
@@ -136,11 +191,11 @@ public class GroupManagerImpl implements GroupManager {
             vo.setTopicName(topicName);
             vo.setPartitionId(groupMetrics.getPartitionId());
 
-            MemberDescription memberDescription = tpMemberMap.get(new TopicPartition(topicName, groupMetrics.getPartitionId()));
+            KSMemberDescription ksMemberDescription = tpMemberMap.get(new TopicPartition(topicName, groupMetrics.getPartitionId()));
-            if (memberDescription != null) {
+            if (ksMemberDescription != null) {
-                vo.setMemberId(memberDescription.consumerId());
+                vo.setMemberId(ksMemberDescription.consumerId());
-                vo.setHost(memberDescription.host());
+                vo.setHost(ksMemberDescription.host());
-                vo.setClientId(memberDescription.clientId());
+                vo.setClientId(ksMemberDescription.clientId());
             }
 
             vo.setLatestMetrics(groupMetrics);
@@ -166,13 +221,18 @@ public class GroupManagerImpl implements GroupManager {
             return rv;
         }
 
-        ConsumerGroupDescription description = groupService.getGroupDescription(dto.getClusterId(), dto.getGroupName());
+        ClusterPhy clusterPhy = clusterPhyService.getClusterByCluster(dto.getClusterId());
+        if (clusterPhy == null) {
+            return Result.buildFromRSAndMsg(ResultStatus.CLUSTER_NOT_EXIST, MsgConstant.getClusterPhyNotExist(dto.getClusterId()));
+        }
+
+        KSGroupDescription description = groupService.getGroupDescriptionFromKafka(clusterPhy, dto.getGroupName());
         if (ConsumerGroupState.DEAD.equals(description.state()) && !dto.isCreateIfNotExist()) {
             return Result.buildFromRSAndMsg(ResultStatus.KAFKA_OPERATE_FAILED, "group不存在, 重置失败");
         }
 
         if (!ConsumerGroupState.EMPTY.equals(description.state()) && !ConsumerGroupState.DEAD.equals(description.state())) {
-            return Result.buildFromRSAndMsg(ResultStatus.KAFKA_OPERATE_FAILED, String.format("group处于%s, 重置失败(仅Empty情况可重置)", GroupStateEnum.getByRawState(description.state()).getState()));
+            return Result.buildFromRSAndMsg(ResultStatus.KAFKA_OPERATE_FAILED, String.format("group处于%s, 重置失败(仅Empty | Dead 情况可重置)", GroupStateEnum.getByRawState(description.state()).getState()));
         }
 
         // 获取offset
@@ -185,6 +245,22 @@ public class GroupManagerImpl implements GroupManager {
         return groupService.resetGroupOffsets(dto.getClusterId(), dto.getGroupName(), offsetMapResult.getData(), operator);
     }
 
+    @Override
+    public List<GroupTopicOverviewVO> getGroupTopicOverviewVOList(Long clusterPhyId, List<GroupMemberPO> groupMemberPOList) {
+        // 获取指标
+        Result<List<GroupMetrics>> metricsListResult = groupMetricService.listLatestMetricsAggByGroupTopicFromES(
+                clusterPhyId,
+                groupMemberPOList.stream().map(elem -> new GroupTopic(elem.getGroupName(), elem.getTopicName())).collect(Collectors.toList()),
+                Arrays.asList(GroupMetricVersionItems.GROUP_METRIC_LAG),
+                AggTypeEnum.MAX
+        );
+        if (metricsListResult.failed()) {
+            // 如果查询失败，则输出错误信息，但是依旧进行已有数据的返回
+            log.error("method=completeMetricData||clusterPhyId={}||result={}||errMsg=search es failed", clusterPhyId, metricsListResult);
+        }
+        return this.convert2GroupTopicOverviewVOList(groupMemberPOList, metricsListResult.getData());
+    }
+
 
     /**************************************************** private method ****************************************************/
 
@@ -221,16 +297,16 @@ public class GroupManagerImpl implements GroupManager {
             )));
         }
 
-        OffsetSpec offsetSpec = null;
+        KSOffsetSpec offsetSpec = null;
         if (OffsetTypeEnum.PRECISE_TIMESTAMP.getResetType() == dto.getResetType()) {
-            offsetSpec = OffsetSpec.forTimestamp(dto.getTimestamp());
+            offsetSpec = KSOffsetSpec.forTimestamp(dto.getTimestamp());
         } else if (OffsetTypeEnum.EARLIEST.getResetType() == dto.getResetType()) {
-            offsetSpec = OffsetSpec.earliest();
+            offsetSpec = KSOffsetSpec.earliest();
         } else {
-            offsetSpec = OffsetSpec.latest();
+            offsetSpec = KSOffsetSpec.latest();
         }
 
-        return partitionService.getPartitionOffsetFromKafka(dto.getClusterId(), dto.getTopicName(), offsetSpec, dto.getTimestamp());
+        return partitionService.getPartitionOffsetFromKafka(dto.getClusterId(), dto.getTopicName(), offsetSpec);
     }
 
     private List<GroupTopicOverviewVO> convert2GroupTopicOverviewVOList(List<GroupMemberPO> poList, List<GroupMetrics> metricsList) {
@@ -272,15 +348,11 @@ public class GroupManagerImpl implements GroupManager {
 
 
         // 获取Group指标信息
-        Result<List<GroupMetrics>> groupMetricsResult = groupMetricService.listPartitionLatestMetricsFromES(
+        Result<List<GroupMetrics>> groupMetricsResult = groupMetricService.collectGroupMetricsFromKafka(clusterPhyId, groupName, latestMetricNames == null ? Arrays.asList() : latestMetricNames);
-                clusterPhyId,
+
-                groupName,
-                topicName,
-                latestMetricNames == null? Arrays.asList(): latestMetricNames
-        );
 
         // 转换Group指标
-        List<GroupMetrics> esGroupMetricsList = groupMetricsResult.hasData()? groupMetricsResult.getData(): new ArrayList<>();
+        List<GroupMetrics> esGroupMetricsList = groupMetricsResult.hasData() ? groupMetricsResult.getData().stream().filter(elem -> topicName.equals(elem.getTopic())).collect(Collectors.toList()) : new ArrayList<>();
         Map<Integer, GroupMetrics> esMetricsMap = new HashMap<>();
         for (GroupMetrics groupMetrics: esGroupMetricsList) {
             esMetricsMap.put(groupMetrics.getPartitionId(), groupMetrics);
@@ -296,5 +368,4 @@ public class GroupManagerImpl implements GroupManager {
                 dto
         );
     }
-
 }

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/reassign/impl/ReassignManagerImpl.java

@@ -22,7 +22,7 @@ import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
 import com.xiaojukeji.know.streaming.km.core.service.reassign.ReassignService;
 import com.xiaojukeji.know.streaming.km.core.service.topic.TopicMetricService;
 import com.xiaojukeji.know.streaming.km.core.service.topic.TopicService;
-import com.xiaojukeji.know.streaming.km.core.service.version.metrics.TopicMetricVersionItems;
+import com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.TopicMetricVersionItems;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Component;
 

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/topic/TopicStateManager.java

@@ -1,8 +1,10 @@
 package com.xiaojukeji.know.streaming.km.biz.topic;
 
-import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationSortDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationBaseDTO;
 import com.xiaojukeji.know.streaming.km.common.bean.dto.topic.TopicRecordDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.group.GroupTopicOverviewVO;
 import com.xiaojukeji.know.streaming.km.common.bean.vo.topic.TopicBrokersPartitionsSummaryVO;
 import com.xiaojukeji.know.streaming.km.common.bean.vo.topic.TopicRecordVO;
 import com.xiaojukeji.know.streaming.km.common.bean.vo.topic.TopicStateVO;
@@ -23,4 +25,6 @@ public interface TopicStateManager {
     Result<List<TopicPartitionVO>> getTopicPartitions(Long clusterPhyId, String topicName, List<String> metricsNames);
 
     Result<TopicBrokersPartitionsSummaryVO> getTopicBrokersPartitionsSummary(Long clusterPhyId, String topicName);
+
+    PaginationResult<GroupTopicOverviewVO> pagingTopicGroupsOverview(Long clusterPhyId, String topicName, String searchGroupName, PaginationBaseDTO dto);
 }

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/topic/impl/OpTopicManagerImpl.java

@@ -10,14 +10,18 @@ import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.param.topic.TopicCreateParam;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.param.topic.TopicParam;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.param.topic.TopicPartitionExpandParam;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.partition.Partition;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.result.ResultStatus;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.topic.Topic;
 import com.xiaojukeji.know.streaming.km.common.constant.MsgConstant;
+import com.xiaojukeji.know.streaming.km.common.utils.BackoffUtils;
+import com.xiaojukeji.know.streaming.km.common.utils.FutureUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
 import com.xiaojukeji.know.streaming.km.common.utils.kafka.KafkaReplicaAssignUtil;
 import com.xiaojukeji.know.streaming.km.core.service.broker.BrokerService;
 import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterPhyService;
+import com.xiaojukeji.know.streaming.km.core.service.partition.PartitionService;
 import com.xiaojukeji.know.streaming.km.core.service.topic.OpTopicService;
 import com.xiaojukeji.know.streaming.km.core.service.topic.TopicService;
 import kafka.admin.AdminUtils;
@@ -52,6 +56,9 @@ public class OpTopicManagerImpl implements OpTopicManager {
     @Autowired
     private ClusterPhyService clusterPhyService;
 
+    @Autowired
+    private PartitionService partitionService;
+
     @Override
     public Result<Void> createTopic(TopicCreateDTO dto, String operator) {
         log.info("method=createTopic||param={}||operator={}.", dto, operator);
@@ -80,7 +87,7 @@ public class OpTopicManagerImpl implements OpTopicManager {
                 );
 
         // 创建Topic
-        return opTopicService.createTopic(
+        Result<Void> createTopicRes = opTopicService.createTopic(
                 new TopicCreateParam(
                         dto.getClusterId(),
                         dto.getTopicName(),
@@ -90,6 +97,21 @@ public class OpTopicManagerImpl implements OpTopicManager {
                 ),
                 operator
         );
+        if (createTopicRes.successful()){
+            try{
+                FutureUtil.quickStartupFutureUtil.submitTask(() -> {
+                    BackoffUtils.backoff(3000);
+                    Result<List<Partition>> partitionsResult = partitionService.listPartitionsFromKafka(clusterPhy, dto.getTopicName());
+                    if (partitionsResult.successful()){
+                        partitionService.updatePartitions(clusterPhy.getId(), dto.getTopicName(), partitionsResult.getData(),  new ArrayList<>());
+                    }
+                });
+            }catch (Exception e) {
+                log.error("method=createTopic||param={}||operator={}||msg=add partition to db failed||errMsg=exception", dto, operator, e);
+                return Result.buildFromRSAndMsg(ResultStatus.MYSQL_OPERATE_FAILED, "Topic创建成功，但记录Partition到DB中失败，等待定时任务同步partition信息");
+            }
+        }
+        return createTopicRes;
     }
 
     @Override

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/topic/impl/TopicConfigManagerImpl.java

@@ -16,7 +16,7 @@ import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
 import com.xiaojukeji.know.streaming.km.core.service.broker.BrokerConfigService;
 import com.xiaojukeji.know.streaming.km.core.service.broker.BrokerService;
 import com.xiaojukeji.know.streaming.km.core.service.topic.TopicConfigService;
-import com.xiaojukeji.know.streaming.km.core.service.version.BaseVersionControlService;
+import com.xiaojukeji.know.streaming.km.core.service.version.BaseKafkaVersionControlService;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Component;
 
@@ -27,7 +27,7 @@ import java.util.stream.Collectors;
 import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionEnum.*;
 
 @Component
-public class TopicConfigManagerImpl extends BaseVersionControlService implements TopicConfigManager {
+public class TopicConfigManagerImpl extends BaseKafkaVersionControlService implements TopicConfigManager {
     private static final ILog log = LogFactory.getLog(TopicConfigManagerImpl.class);
 
     private static final String GET_DEFAULT_TOPIC_CONFIG       = "getDefaultTopicConfig";

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/topic/impl/TopicStateManagerImpl.java

@@ -2,17 +2,23 @@ package com.xiaojukeji.know.streaming.km.biz.topic.impl;
 
 import com.didiglobal.logi.log.ILog;
 import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.biz.group.GroupManager;
 import com.xiaojukeji.know.streaming.km.biz.topic.TopicStateManager;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationBaseDTO;
 import com.xiaojukeji.know.streaming.km.common.bean.dto.topic.TopicRecordDTO;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.broker.Broker;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.PartitionMetrics;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.TopicMetrics;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.offset.KSOffsetSpec;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.partition.Partition;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.result.ResultStatus;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.topic.Topic;
+import com.xiaojukeji.know.streaming.km.common.bean.po.group.GroupMemberPO;
 import com.xiaojukeji.know.streaming.km.common.bean.vo.broker.BrokerReplicaSummaryVO;
+import com.xiaojukeji.know.streaming.km.common.bean.vo.group.GroupTopicOverviewVO;
 import com.xiaojukeji.know.streaming.km.common.bean.vo.topic.TopicBrokersPartitionsSummaryVO;
 import com.xiaojukeji.know.streaming.km.common.bean.vo.topic.TopicRecordVO;
 import com.xiaojukeji.know.streaming.km.common.bean.vo.topic.TopicStateVO;
@@ -32,15 +38,15 @@ import com.xiaojukeji.know.streaming.km.common.utils.PaginationUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
 import com.xiaojukeji.know.streaming.km.core.service.broker.BrokerService;
 import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterPhyService;
+import com.xiaojukeji.know.streaming.km.core.service.group.GroupService;
 import com.xiaojukeji.know.streaming.km.core.service.partition.PartitionMetricService;
 import com.xiaojukeji.know.streaming.km.core.service.partition.PartitionService;
 import com.xiaojukeji.know.streaming.km.core.service.topic.TopicConfigService;
 import com.xiaojukeji.know.streaming.km.core.service.topic.TopicMetricService;
 import com.xiaojukeji.know.streaming.km.core.service.topic.TopicService;
-import com.xiaojukeji.know.streaming.km.core.service.version.metrics.TopicMetricVersionItems;
+import com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.TopicMetricVersionItems;
 import org.apache.commons.lang3.ObjectUtils;
 import org.apache.commons.lang3.StringUtils;
-import org.apache.kafka.clients.admin.OffsetSpec;
 import org.apache.kafka.clients.consumer.*;
 import org.apache.kafka.common.TopicPartition;
 import org.apache.kafka.common.config.TopicConfig;
@@ -77,6 +83,12 @@ public class TopicStateManagerImpl implements TopicStateManager {
     @Autowired
     private TopicConfigService topicConfigService;
 
+    @Autowired
+    private GroupService groupService;
+
+    @Autowired
+    private GroupManager groupManager;
+
     @Override
     public TopicBrokerAllVO getTopicBrokerAll(Long clusterPhyId, String topicName, String searchBrokerHost) throws NotExistException {
         Topic topic = topicService.getTopic(clusterPhyId, topicName);
@@ -131,12 +143,12 @@ public class TopicStateManagerImpl implements TopicStateManager {
         }
 
         // 获取分区beginOffset
-        Result<Map<TopicPartition, Long>> beginOffsetsMapResult = partitionService.getPartitionOffsetFromKafka(clusterPhyId, topicName, dto.getFilterPartitionId(), OffsetSpec.earliest(), null);
+        Result<Map<TopicPartition, Long>> beginOffsetsMapResult = partitionService.getPartitionOffsetFromKafka(clusterPhyId, topicName, dto.getFilterPartitionId(), KSOffsetSpec.earliest());
         if (beginOffsetsMapResult.failed()) {
             return Result.buildFromIgnoreData(beginOffsetsMapResult);
         }
         // 获取分区endOffset
-        Result<Map<TopicPartition, Long>> endOffsetsMapResult = partitionService.getPartitionOffsetFromKafka(clusterPhyId, topicName, dto.getFilterPartitionId(), OffsetSpec.latest(), null);
+        Result<Map<TopicPartition, Long>> endOffsetsMapResult = partitionService.getPartitionOffsetFromKafka(clusterPhyId, topicName, dto.getFilterPartitionId(), KSOffsetSpec.latest());
         if (endOffsetsMapResult.failed()) {
             return Result.buildFromIgnoreData(endOffsetsMapResult);
         }
@@ -295,7 +307,7 @@ public class TopicStateManagerImpl implements TopicStateManager {
         if (metricsResult.failed()) {
             // 仅打印错误日志，但是不直接返回错误
             log.error(
-                    "class=TopicStateManagerImpl||method=getTopicPartitions||clusterPhyId={}||topicName={}||result={}||msg=get metrics from es failed",
+                    "method=getTopicPartitions||clusterPhyId={}||topicName={}||result={}||msg=get metrics from es failed",
                     clusterPhyId, topicName, metricsResult
             );
         }
@@ -346,6 +358,19 @@ public class TopicStateManagerImpl implements TopicStateManager {
         return Result.buildSuc(vo);
     }
 
+    @Override
+    public PaginationResult<GroupTopicOverviewVO> pagingTopicGroupsOverview(Long clusterPhyId, String topicName, String searchGroupName, PaginationBaseDTO dto) {
+        PaginationResult<GroupMemberPO> paginationResult = groupService.pagingGroupMembers(clusterPhyId, topicName, "", "", searchGroupName, dto);
+
+        if (!paginationResult.hasData()) {
+            return PaginationResult.buildSuc(new ArrayList<>(), paginationResult);
+        }
+
+        List<GroupTopicOverviewVO> groupTopicVOList = groupManager.getGroupTopicOverviewVOList(clusterPhyId, paginationResult.getData().getBizData());
+
+        return PaginationResult.buildSuc(groupTopicVOList, paginationResult);
+    }
+
     /**************************************************** private method ****************************************************/
 
     private boolean checkIfIgnore(ConsumerRecord<String, String> consumerRecord, String filterKey, String filterValue) {

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/version/VersionControlManager.java

@@ -20,7 +20,7 @@ public interface VersionControlManager {
      * 获取当前ks所有支持的kafka版本
      * @return
      */
-    Result<Map<String, Long>> listAllVersions();
+    Result<Map<String, Long>> listAllKafkaVersions();
 
     /**
      * 获取全部集群 clusterId 中类型为 type 的指标，不论支持不支持
@@ -28,7 +28,7 @@ public interface VersionControlManager {
      * @param type
      * @return
      */
-    Result<List<VersionItemVO>> listClusterVersionControlItem(Long clusterId, Integer type);
+    Result<List<VersionItemVO>> listKafkaClusterVersionControlItem(Long clusterId, Integer type);
 
     /**
      * 获取当前用户设置的用于展示的指标配置

km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz/version/impl/VersionControlManagerImpl.java

@@ -14,10 +14,10 @@ import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionControlItem;
 import com.xiaojukeji.know.streaming.km.common.bean.vo.config.metric.UserMetricConfigVO;
 import com.xiaojukeji.know.streaming.km.common.bean.vo.version.VersionItemVO;
-import com.xiaojukeji.know.streaming.km.common.constant.Constant;
 import com.xiaojukeji.know.streaming.km.common.enums.version.VersionEnum;
 import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.VersionUtil;
+import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterPhyService;
 import com.xiaojukeji.know.streaming.km.core.service.version.VersionControlService;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Service;
@@ -30,10 +30,12 @@ import java.util.stream.Collectors;
 
 import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionEnum.V_MAX;
 import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum.*;
-import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.BrokerMetricVersionItems.*;
+import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.BrokerMetricVersionItems.*;
-import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.ClusterMetricVersionItems.*;
+import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.ClusterMetricVersionItems.*;
-import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.GroupMetricVersionItems.*;
+import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.GroupMetricVersionItems.*;
-import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.TopicMetricVersionItems.*;
+import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.TopicMetricVersionItems.*;
+import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.connect.MirrorMakerMetricVersionItems.*;
+import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.ZookeeperMetricVersionItems.*;
 
 @Service
 public class VersionControlManagerImpl implements VersionControlManager {
@@ -48,7 +50,8 @@ public class VersionControlManagerImpl implements VersionControlManager {
 
     @PostConstruct
     public void init(){
-        defaultMetrics.add(new UserMetricConfig(METRIC_TOPIC.getCode(), TOPIC_METRIC_HEALTH_SCORE, true));
+        // topic
+        defaultMetrics.add(new UserMetricConfig(METRIC_TOPIC.getCode(), TOPIC_METRIC_HEALTH_STATE, true));
         defaultMetrics.add(new UserMetricConfig(METRIC_TOPIC.getCode(), TOPIC_METRIC_FAILED_FETCH_REQ, true));
         defaultMetrics.add(new UserMetricConfig(METRIC_TOPIC.getCode(), TOPIC_METRIC_FAILED_PRODUCE_REQ, true));
         defaultMetrics.add(new UserMetricConfig(METRIC_TOPIC.getCode(), TOPIC_METRIC_UNDER_REPLICA_PARTITIONS, true));
@@ -58,7 +61,8 @@ public class VersionControlManagerImpl implements VersionControlManager {
         defaultMetrics.add(new UserMetricConfig(METRIC_TOPIC.getCode(), TOPIC_METRIC_BYTES_REJECTED, true));
         defaultMetrics.add(new UserMetricConfig(METRIC_TOPIC.getCode(), TOPIC_METRIC_MESSAGE_IN, true));
 
-        defaultMetrics.add(new UserMetricConfig(METRIC_CLUSTER.getCode(), CLUSTER_METRIC_HEALTH_SCORE, true));
+        // cluster
+        defaultMetrics.add(new UserMetricConfig(METRIC_CLUSTER.getCode(), CLUSTER_METRIC_HEALTH_STATE, true));
         defaultMetrics.add(new UserMetricConfig(METRIC_CLUSTER.getCode(), CLUSTER_METRIC_ACTIVE_CONTROLLER_COUNT, true));
         defaultMetrics.add(new UserMetricConfig(METRIC_CLUSTER.getCode(), CLUSTER_METRIC_BYTES_IN, true));
         defaultMetrics.add(new UserMetricConfig(METRIC_CLUSTER.getCode(), CLUSTER_METRIC_BYTES_OUT, true));
@@ -73,12 +77,14 @@ public class VersionControlManagerImpl implements VersionControlManager {
         defaultMetrics.add(new UserMetricConfig(METRIC_CLUSTER.getCode(), CLUSTER_METRIC_GROUP_REBALANCES, true));
         defaultMetrics.add(new UserMetricConfig(METRIC_CLUSTER.getCode(), CLUSTER_METRIC_JOB_RUNNING, true));
 
+        // group
         defaultMetrics.add(new UserMetricConfig(METRIC_GROUP.getCode(), GROUP_METRIC_OFFSET_CONSUMED, true));
         defaultMetrics.add(new UserMetricConfig(METRIC_GROUP.getCode(), GROUP_METRIC_LAG, true));
         defaultMetrics.add(new UserMetricConfig(METRIC_GROUP.getCode(), GROUP_METRIC_STATE, true));
-        defaultMetrics.add(new UserMetricConfig(METRIC_GROUP.getCode(), GROUP_METRIC_HEALTH_SCORE, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_GROUP.getCode(), GROUP_METRIC_HEALTH_STATE, true));
 
-        defaultMetrics.add(new UserMetricConfig(METRIC_BROKER.getCode(), BROKER_METRIC_HEALTH_SCORE, true));
+        // broker
+        defaultMetrics.add(new UserMetricConfig(METRIC_BROKER.getCode(), BROKER_METRIC_HEALTH_STATE, true));
         defaultMetrics.add(new UserMetricConfig(METRIC_BROKER.getCode(), BROKER_METRIC_CONNECTION_COUNT, true));
         defaultMetrics.add(new UserMetricConfig(METRIC_BROKER.getCode(), BROKER_METRIC_MESSAGE_IN, true));
         defaultMetrics.add(new UserMetricConfig(METRIC_BROKER.getCode(), BROKER_METRIC_NETWORK_RPO_AVG_IDLE, true));
@@ -91,8 +97,37 @@ public class VersionControlManagerImpl implements VersionControlManager {
         defaultMetrics.add(new UserMetricConfig(METRIC_BROKER.getCode(), BROKER_METRIC_PARTITIONS_SKEW, true));
         defaultMetrics.add(new UserMetricConfig(METRIC_BROKER.getCode(), BROKER_METRIC_BYTES_IN, true));
         defaultMetrics.add(new UserMetricConfig(METRIC_BROKER.getCode(), BROKER_METRIC_BYTES_OUT, true));
+
+        // zookeeper
+        defaultMetrics.add(new UserMetricConfig(METRIC_ZOOKEEPER.getCode(), ZOOKEEPER_METRIC_HEALTH_STATE, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_ZOOKEEPER.getCode(), ZOOKEEPER_METRIC_HEALTH_CHECK_PASSED, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_ZOOKEEPER.getCode(), ZOOKEEPER_METRIC_HEALTH_CHECK_TOTAL, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_ZOOKEEPER.getCode(), ZOOKEEPER_METRIC_MAX_REQUEST_LATENCY, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_ZOOKEEPER.getCode(), ZOOKEEPER_METRIC_OUTSTANDING_REQUESTS, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_ZOOKEEPER.getCode(), ZOOKEEPER_METRIC_NODE_COUNT, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_ZOOKEEPER.getCode(), ZOOKEEPER_METRIC_WATCH_COUNT, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_ZOOKEEPER.getCode(), ZOOKEEPER_METRIC_NUM_ALIVE_CONNECTIONS, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_ZOOKEEPER.getCode(), ZOOKEEPER_METRIC_PACKETS_RECEIVED, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_ZOOKEEPER.getCode(), ZOOKEEPER_METRIC_PACKETS_SENT, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_ZOOKEEPER.getCode(), ZOOKEEPER_METRIC_EPHEMERALS_COUNT, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_ZOOKEEPER.getCode(), ZOOKEEPER_METRIC_APPROXIMATE_DATA_SIZE, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_ZOOKEEPER.getCode(), ZOOKEEPER_METRIC_OPEN_FILE_DESCRIPTOR_COUNT, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_ZOOKEEPER.getCode(), ZOOKEEPER_METRIC_KAFKA_ZK_DISCONNECTS_PER_SEC, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_ZOOKEEPER.getCode(), ZOOKEEPER_METRIC_KAFKA_ZK_SYNC_CONNECTS_PER_SEC, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_ZOOKEEPER.getCode(), ZOOKEEPER_METRIC_KAFKA_ZK_REQUEST_LATENCY_99TH, true));
+
+        // mm2
+        defaultMetrics.add(new UserMetricConfig(METRIC_CONNECT_MIRROR_MAKER.getCode(), MIRROR_MAKER_METRIC_BYTE_COUNT, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_CONNECT_MIRROR_MAKER.getCode(), MIRROR_MAKER_METRIC_BYTE_RATE, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_CONNECT_MIRROR_MAKER.getCode(), MIRROR_MAKER_METRIC_RECORD_AGE_MS_MAX, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_CONNECT_MIRROR_MAKER.getCode(), MIRROR_MAKER_METRIC_RECORD_COUNT, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_CONNECT_MIRROR_MAKER.getCode(), MIRROR_MAKER_METRIC_RECORD_RATE, true));
+        defaultMetrics.add(new UserMetricConfig(METRIC_CONNECT_MIRROR_MAKER.getCode(), MIRROR_MAKER_METRIC_REPLICATION_LATENCY_MS_MAX, true));
     }
 
+    @Autowired
+    private ClusterPhyService clusterPhyService;
+
     @Autowired
     private VersionControlService versionControlService;
 
@@ -108,27 +143,40 @@ public class VersionControlManagerImpl implements VersionControlManager {
         allVersionItemVO.addAll(ConvertUtil.list2List(versionControlService.listVersionControlItem(METRIC_BROKER.getCode()), VersionItemVO.class));
         allVersionItemVO.addAll(ConvertUtil.list2List(versionControlService.listVersionControlItem(METRIC_PARTITION.getCode()), VersionItemVO.class));
         allVersionItemVO.addAll(ConvertUtil.list2List(versionControlService.listVersionControlItem(METRIC_REPLICATION.getCode()), VersionItemVO.class));
+
+        allVersionItemVO.addAll(ConvertUtil.list2List(versionControlService.listVersionControlItem(METRIC_ZOOKEEPER.getCode()), VersionItemVO.class));
+
+        allVersionItemVO.addAll(ConvertUtil.list2List(versionControlService.listVersionControlItem(METRIC_CONNECT_CLUSTER.getCode()), VersionItemVO.class));
+        allVersionItemVO.addAll(ConvertUtil.list2List(versionControlService.listVersionControlItem(METRIC_CONNECT_CONNECTOR.getCode()), VersionItemVO.class));
+        allVersionItemVO.addAll(ConvertUtil.list2List(versionControlService.listVersionControlItem(METRIC_CONNECT_MIRROR_MAKER.getCode()), VersionItemVO.class));
+
         allVersionItemVO.addAll(ConvertUtil.list2List(versionControlService.listVersionControlItem(WEB_OP.getCode()), VersionItemVO.class));
 
         Map<String, VersionItemVO> map = allVersionItemVO.stream().collect(
-                Collectors.toMap(u -> u.getType() + "@" + u.getName(), Function.identity() ));
+                Collectors.toMap(
+                        u -> u.getType() + "@" + u.getName(),
+                        Function.identity(),
+                        (v1, v2) -> v1)
+        );
 
         return Result.buildSuc(map);
     }
 
     @Override
-    public Result<Map<String, Long>> listAllVersions() {
+    public Result<Map<String, Long>> listAllKafkaVersions() {
         return Result.buildSuc(VersionEnum.allVersionsWithOutMax());
     }
 
     @Override
-    public Result<List<VersionItemVO>> listClusterVersionControlItem(Long clusterId, Integer type) {
+    public Result<List<VersionItemVO>> listKafkaClusterVersionControlItem(Long clusterId, Integer type) {
         List<VersionControlItem> allItem   = versionControlService.listVersionControlItem(type);
         List<VersionItemVO> versionItemVOS = new ArrayList<>();
 
+        String versionStr  = clusterPhyService.getVersionFromCacheFirst(clusterId);
+
         for (VersionControlItem item : allItem){
             VersionItemVO itemVO = ConvertUtil.obj2Obj(item, VersionItemVO.class);
-            boolean      support = versionControlService.isClusterSupport(clusterId, item);
+            boolean      support = versionControlService.isClusterSupport(versionStr, item);
 
             itemVO.setSupport(support);
             itemVO.setDesc(itemSupportDesc(item, support));
@@ -141,7 +189,7 @@ public class VersionControlManagerImpl implements VersionControlManager {
 
     @Override
     public Result<List<UserMetricConfigVO>> listUserMetricItem(Long clusterId, Integer type, String operator) {
-        Result<List<VersionItemVO>> ret = listClusterVersionControlItem(clusterId, type);
+        Result<List<VersionItemVO>> ret = listKafkaClusterVersionControlItem(clusterId, type);
         if(null == ret || ret.failed()){
             return Result.buildFail();
         }

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/metric/AbstractMetricCollector.java

@@ -1,7 +1,6 @@
 package com.xiaojukeji.know.streaming.km.collector.metric;
 
 import com.xiaojukeji.know.streaming.km.collector.service.CollectThreadPoolService;
-import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
 import com.xiaojukeji.know.streaming.km.common.bean.event.metric.BaseMetricEvent;
 import com.xiaojukeji.know.streaming.km.common.component.SpringTool;
 import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
@@ -9,17 +8,20 @@ import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
 import org.springframework.beans.factory.annotation.Autowired;
 
 
+
 /**
  * @author didi
  */
-public abstract class AbstractMetricCollector<T> {
+public abstract class AbstractMetricCollector<M, C> {
-    public abstract void collectMetrics(ClusterPhy clusterPhy);
+    public abstract String getClusterVersion(C c);
 
     public abstract VersionItemTypeEnum collectorType();
 
     @Autowired
     private CollectThreadPoolService collectThreadPoolService;
 
+    public abstract void collectMetrics(C c);
+
     protected FutureWaitUtil<Void> getFutureUtilByClusterPhyId(Long clusterPhyId) {
         return collectThreadPoolService.selectSuitableFutureUtil(clusterPhyId * 1000L + this.collectorType().getCode());
     }

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/metric/MetricESSender.java

@@ -1,121 +0,0 @@
-package com.xiaojukeji.know.streaming.km.collector.metric;
-
-import com.didiglobal.logi.log.ILog;
-import com.didiglobal.logi.log.LogFactory;
-import com.xiaojukeji.know.streaming.km.common.bean.event.metric.*;
-import com.xiaojukeji.know.streaming.km.common.bean.po.BaseESPO;
-import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.*;
-import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
-import com.xiaojukeji.know.streaming.km.common.utils.EnvUtil;
-import com.xiaojukeji.know.streaming.km.common.utils.NamedThreadFactory;
-import com.xiaojukeji.know.streaming.km.persistence.es.dao.BaseMetricESDAO;
-import org.apache.commons.collections.CollectionUtils;
-import org.springframework.context.ApplicationListener;
-import org.springframework.stereotype.Component;
-
-import javax.annotation.PostConstruct;
-import java.util.List;
-import java.util.Objects;
-import java.util.concurrent.LinkedBlockingDeque;
-import java.util.concurrent.ThreadPoolExecutor;
-import java.util.concurrent.TimeUnit;
-
-import static com.xiaojukeji.know.streaming.km.common.constant.ESIndexConstant.*;
-
-@Component
-public class MetricESSender implements ApplicationListener<BaseMetricEvent> {
-    protected static final ILog  LOGGER = LogFactory.getLog("METRIC_LOGGER");
-
-    private static final int    THRESHOLD = 100;
-
-    private ThreadPoolExecutor esExecutor = new ThreadPoolExecutor(10, 20, 6000, TimeUnit.MILLISECONDS,
-            new LinkedBlockingDeque<>(1000),
-            new NamedThreadFactory("KM-Collect-MetricESSender-ES"),
-            (r, e) -> LOGGER.warn("class=MetricESSender||msg=KM-Collect-MetricESSender-ES Deque is blocked, taskCount:{}" + e.getTaskCount()));
-
-    @PostConstruct
-    public void init(){
-        LOGGER.info("class=MetricESSender||method=init||msg=init finished");
-    }
-
-    @Override
-    public void onApplicationEvent(BaseMetricEvent event) {
-        if(event instanceof BrokerMetricEvent) {
-            BrokerMetricEvent brokerMetricEvent = (BrokerMetricEvent)event;
-            send2es(BROKER_INDEX,
-                    ConvertUtil.list2List(brokerMetricEvent.getBrokerMetrics(), BrokerMetricPO.class)
-            );
-
-        } else if(event instanceof ClusterMetricEvent) {
-            ClusterMetricEvent clusterMetricEvent = (ClusterMetricEvent)event;
-            send2es(CLUSTER_INDEX,
-                    ConvertUtil.list2List(clusterMetricEvent.getClusterMetrics(), ClusterMetricPO.class)
-            );
-
-        } else if(event instanceof TopicMetricEvent) {
-            TopicMetricEvent topicMetricEvent = (TopicMetricEvent)event;
-            send2es(TOPIC_INDEX,
-                    ConvertUtil.list2List(topicMetricEvent.getTopicMetrics(), TopicMetricPO.class)
-            );
-
-        } else if(event instanceof PartitionMetricEvent) {
-            PartitionMetricEvent partitionMetricEvent = (PartitionMetricEvent)event;
-            send2es(PARTITION_INDEX,
-                    ConvertUtil.list2List(partitionMetricEvent.getPartitionMetrics(), PartitionMetricPO.class)
-            );
-
-        } else if(event instanceof GroupMetricEvent) {
-            GroupMetricEvent groupMetricEvent = (GroupMetricEvent)event;
-            send2es(GROUP_INDEX,
-                    ConvertUtil.list2List(groupMetricEvent.getGroupMetrics(), GroupMetricPO.class)
-            );
-
-        } else if(event instanceof ReplicaMetricEvent) {
-            ReplicaMetricEvent replicaMetricEvent = (ReplicaMetricEvent)event;
-            send2es(REPLICATION_INDEX,
-                    ConvertUtil.list2List(replicaMetricEvent.getReplicationMetrics(), ReplicationMetricPO.class)
-            );
-        }
-    }
-
-    /**
-     * 根据不同监控维度来发送
-     */
-    private boolean send2es(String index, List<? extends BaseESPO> statsList){
-        if (CollectionUtils.isEmpty(statsList)) {
-            return true;
-        }
-
-        if (!EnvUtil.isOnline()) {
-            LOGGER.info("class=MetricESSender||method=send2es||ariusStats={}||size={}",
-                    index, statsList.size());
-        }
-
-        BaseMetricESDAO baseMetricESDao = BaseMetricESDAO.getByStatsType(index);
-        if (Objects.isNull( baseMetricESDao )) {
-            LOGGER.error("class=MetricESSender||method=send2es||errMsg=fail to find {}", index);
-            return false;
-        }
-
-        int size = statsList.size();
-        int num  = (size) % THRESHOLD == 0 ? (size / THRESHOLD) : (size / THRESHOLD + 1);
-
-        if (size < THRESHOLD) {
-            esExecutor.execute(
-                    () -> baseMetricESDao.batchInsertStats(statsList)
-            );
-            return true;
-        }
-
-        for (int i = 1; i < num + 1; i++) {
-            int end   = (i * THRESHOLD) > size ? size : (i * THRESHOLD);
-            int start = (i - 1) * THRESHOLD;
-
-            esExecutor.execute(
-                    () -> baseMetricESDao.batchInsertStats(statsList.subList(start, end))
-            );
-        }
-
-        return true;
-    }
-}

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/metric/ReplicaMetricCollector.java

@@ -1,124 +0,0 @@
-package com.xiaojukeji.know.streaming.km.collector.metric;
-
-import com.alibaba.fastjson.JSON;
-import com.didiglobal.logi.log.ILog;
-import com.didiglobal.logi.log.LogFactory;
-import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
-import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.ReplicationMetrics;
-import com.xiaojukeji.know.streaming.km.common.bean.entity.partition.Partition;
-import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
-import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionControlItem;
-import com.xiaojukeji.know.streaming.km.common.bean.event.metric.ReplicaMetricEvent;
-import com.xiaojukeji.know.streaming.km.common.constant.Constant;
-import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
-import com.xiaojukeji.know.streaming.km.common.utils.EnvUtil;
-import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
-import com.xiaojukeji.know.streaming.km.core.service.partition.PartitionService;
-import com.xiaojukeji.know.streaming.km.core.service.replica.ReplicaMetricService;
-import com.xiaojukeji.know.streaming.km.core.service.version.VersionControlService;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.stereotype.Component;
-
-import java.util.ArrayList;
-import java.util.List;
-
-import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum.METRIC_REPLICATION;
-
-/**
- * @author didi
- */
-@Component
-public class ReplicaMetricCollector extends AbstractMetricCollector<ReplicationMetrics> {
-    protected static final ILog  LOGGER = LogFactory.getLog("METRIC_LOGGER");
-
-    @Autowired
-    private VersionControlService versionControlService;
-
-    @Autowired
-    private ReplicaMetricService replicaMetricService;
-
-    @Autowired
-    private PartitionService partitionService;
-
-    @Override
-    public void collectMetrics(ClusterPhy clusterPhy) {
-        Long        startTime           =   System.currentTimeMillis();
-        Long        clusterPhyId        =   clusterPhy.getId();
-        List<VersionControlItem> items  =   versionControlService.listVersionControlItem(clusterPhyId, collectorType().getCode());
-
-        List<Partition> partitions = partitionService.listPartitionByCluster(clusterPhyId);
-
-        FutureWaitUtil<Void> future = this.getFutureUtilByClusterPhyId(clusterPhyId);
-
-        List<ReplicationMetrics> metricsList = new ArrayList<>();
-        for(Partition partition : partitions) {
-            for (Integer brokerId: partition.getAssignReplicaList()) {
-                ReplicationMetrics metrics = new ReplicationMetrics(clusterPhyId, partition.getTopicName(), brokerId, partition.getPartitionId());
-                metricsList.add(metrics);
-
-                future.runnableTask(
-                        String.format("method=ReplicaMetricCollector||clusterPhyId=%d||brokerId=%d||topicName=%s||partitionId=%d",
-                                clusterPhyId, brokerId, partition.getTopicName(), partition.getPartitionId()),
-                        30000,
-                        () -> collectMetrics(clusterPhyId, metrics, items)
-                );
-            }
-        }
-
-        future.waitExecute(30000);
-
-        publishMetric(new ReplicaMetricEvent(this, metricsList));
-
-        LOGGER.info("method=ReplicaMetricCollector||clusterPhyId={}||startTime={}||costTime={}||msg=collect finished.",
-                clusterPhyId, startTime, System.currentTimeMillis() - startTime);
-    }
-
-    @Override
-    public VersionItemTypeEnum collectorType() {
-        return METRIC_REPLICATION;
-    }
-
-    /**************************************************** private method ****************************************************/
-
-    private ReplicationMetrics collectMetrics(Long clusterPhyId, ReplicationMetrics metrics, List<VersionControlItem> items) {
-        long startTime = System.currentTimeMillis();
-
-        metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
-
-        for(VersionControlItem v : items) {
-            try {
-                if (metrics.getMetrics().containsKey(v.getName())) {
-                    continue;
-                }
-
-                Result<ReplicationMetrics> ret = replicaMetricService.collectReplicaMetricsFromKafkaWithCache(
-                        clusterPhyId,
-                        metrics.getTopic(),
-                        metrics.getBrokerId(),
-                        metrics.getPartitionId(),
-                        v.getName()
-                );
-
-                if (null == ret || ret.failed() || null == ret.getData()) {
-                    continue;
-                }
-
-                metrics.putMetric(ret.getData().getMetrics());
-
-                if (!EnvUtil.isOnline()) {
-                    LOGGER.info("method=ReplicaMetricCollector||clusterPhyId={}||topicName={}||partitionId={}||metricName={}||metricValue={}",
-                            clusterPhyId, metrics.getTopic(), metrics.getPartitionId(), v.getName(), JSON.toJSONString(ret.getData().getMetrics()));
-                }
-
-            } catch (Exception e) {
-                LOGGER.error("method=ReplicaMetricCollector||clusterPhyId={}||topicName={}||partition={}||metricName={}||errMsg=exception!",
-                        clusterPhyId, metrics.getTopic(), metrics.getPartitionId(), v.getName(), e);
-            }
-        }
-
-        // 记录采集性能
-        metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, (System.currentTimeMillis() - startTime) / 1000.0f);
-
-        return metrics;
-    }
-}

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/metric/connect/AbstractConnectMetricCollector.java

@@ -0,0 +1,50 @@
+package com.xiaojukeji.know.streaming.km.collector.metric.connect;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.collector.metric.AbstractMetricCollector;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.ConnectCluster;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
+import com.xiaojukeji.know.streaming.km.common.utils.LoggerUtil;
+import com.xiaojukeji.know.streaming.km.core.service.connect.cluster.ConnectClusterService;
+import org.springframework.beans.factory.annotation.Autowired;
+
+import java.util.List;
+
+/**
+ * @author didi
+ */
+public abstract class AbstractConnectMetricCollector<M> extends AbstractMetricCollector<M, ConnectCluster> {
+    private static final ILog LOGGER                    = LogFactory.getLog(AbstractConnectMetricCollector.class);
+
+    protected static final ILog METRIC_COLLECTED_LOGGER = LoggerUtil.getMetricCollectedLogger();
+
+    @Autowired
+    private ConnectClusterService connectClusterService;
+
+    public abstract List<M> collectConnectMetrics(ConnectCluster connectCluster);
+
+    @Override
+    public String getClusterVersion(ConnectCluster connectCluster){
+        return connectClusterService.getClusterVersion(connectCluster.getId());
+    }
+
+    @Override
+    public void collectMetrics(ConnectCluster connectCluster) {
+        long startTime = System.currentTimeMillis();
+
+        // 采集指标
+        List<M> metricsList = this.collectConnectMetrics(connectCluster);
+
+        // 输出耗时信息
+        LOGGER.info(
+                "metricType={}||connectClusterId={}||costTimeUnitMs={}",
+                this.collectorType().getMessage(), connectCluster.getId(), System.currentTimeMillis() - startTime
+        );
+
+        // 输出采集到的指标信息
+        METRIC_COLLECTED_LOGGER.debug("metricType={}||connectClusterId={}||metrics={}!",
+                this.collectorType().getMessage(), connectCluster.getId(), ConvertUtil.obj2Json(metricsList)
+        );
+    }
+}

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/metric/connect/ConnectClusterMetricCollector.java

@@ -0,0 +1,83 @@
+package com.xiaojukeji.know.streaming.km.collector.metric.connect;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.ConnectCluster;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.connect.ConnectClusterMetrics;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionControlItem;
+import com.xiaojukeji.know.streaming.km.common.bean.event.metric.connect.ConnectClusterMetricEvent;
+import com.xiaojukeji.know.streaming.km.common.constant.Constant;
+import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
+import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
+import com.xiaojukeji.know.streaming.km.core.service.connect.cluster.ConnectClusterMetricService;
+import com.xiaojukeji.know.streaming.km.core.service.version.VersionControlService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import java.util.Collections;
+import java.util.List;
+
+import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum.METRIC_CONNECT_CLUSTER;
+
+/**
+ * @author didi
+ */
+@Component
+public class ConnectClusterMetricCollector extends AbstractConnectMetricCollector<ConnectClusterMetrics> {
+    protected static final ILog LOGGER = LogFactory.getLog(ConnectClusterMetricCollector.class);
+
+    @Autowired
+    private VersionControlService versionControlService;
+
+    @Autowired
+    private ConnectClusterMetricService connectClusterMetricService;
+
+    @Override
+    public List<ConnectClusterMetrics> collectConnectMetrics(ConnectCluster connectCluster) {
+        Long startTime          = System.currentTimeMillis();
+        Long clusterPhyId       = connectCluster.getKafkaClusterPhyId();
+        Long connectClusterId   = connectCluster.getId();
+
+        ConnectClusterMetrics metrics = new ConnectClusterMetrics(clusterPhyId, connectClusterId);
+        metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
+        List<VersionControlItem> items = versionControlService.listVersionControlItem(getClusterVersion(connectCluster), collectorType().getCode());
+        FutureWaitUtil<Void> future = this.getFutureUtilByClusterPhyId(connectClusterId);
+
+        for (VersionControlItem item : items) {
+            future.runnableTask(
+                    String.format("class=ConnectClusterMetricCollector||connectClusterId=%d||metricName=%s", connectClusterId, item.getName()),
+                    30000,
+                    () -> {
+                        try {
+                            Result<ConnectClusterMetrics> ret = connectClusterMetricService.collectConnectClusterMetricsFromKafka(connectClusterId, item.getName());
+                            if (null == ret || !ret.hasData()) {
+                                return null;
+                            }
+                            metrics.putMetric(ret.getData().getMetrics());
+
+                        } catch (Exception e) {
+                            LOGGER.error(
+                                    "method=collectConnectMetrics||connectClusterId={}||metricName={}||errMsg=exception!",
+                                    connectClusterId, item.getName(), e
+                            );
+                        }
+                        return null;
+                    }
+            );
+        }
+
+        future.waitExecute(30000);
+
+        metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, (System.currentTimeMillis() - startTime) / 1000.0f);
+
+        this.publishMetric(new ConnectClusterMetricEvent(this, Collections.singletonList(metrics)));
+
+        return Collections.singletonList(metrics);
+    }
+
+    @Override
+    public VersionItemTypeEnum collectorType() {
+        return METRIC_CONNECT_CLUSTER;
+    }
+}

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/metric/connect/ConnectConnectorMetricCollector.java

@@ -0,0 +1,107 @@
+package com.xiaojukeji.know.streaming.km.collector.metric.connect;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.ConnectCluster;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.connect.ConnectorMetrics;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionControlItem;
+import com.xiaojukeji.know.streaming.km.common.bean.event.metric.connect.ConnectorMetricEvent;
+import com.xiaojukeji.know.streaming.km.common.constant.Constant;
+import com.xiaojukeji.know.streaming.km.common.enums.connect.ConnectorTypeEnum;
+import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
+import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
+import com.xiaojukeji.know.streaming.km.core.service.connect.connector.ConnectorMetricService;
+import com.xiaojukeji.know.streaming.km.core.service.connect.connector.ConnectorService;
+import com.xiaojukeji.know.streaming.km.core.service.version.VersionControlService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum.METRIC_CONNECT_CONNECTOR;
+
+/**
+ * @author didi
+ */
+@Component
+public class ConnectConnectorMetricCollector extends AbstractConnectMetricCollector<ConnectorMetrics> {
+    protected static final ILog LOGGER = LogFactory.getLog(ConnectConnectorMetricCollector.class);
+
+    @Autowired
+    private VersionControlService versionControlService;
+
+    @Autowired
+    private ConnectorService connectorService;
+
+    @Autowired
+    private ConnectorMetricService connectorMetricService;
+
+    @Override
+    public List<ConnectorMetrics> collectConnectMetrics(ConnectCluster connectCluster) {
+        Long clusterPhyId       = connectCluster.getKafkaClusterPhyId();
+        Long connectClusterId   = connectCluster.getId();
+
+        List<VersionControlItem> items      = versionControlService.listVersionControlItem(this.getClusterVersion(connectCluster), collectorType().getCode());
+        Result<List<String>> connectorList  = connectorService.listConnectorsFromCluster(connectClusterId);
+
+        FutureWaitUtil<Void> future         = this.getFutureUtilByClusterPhyId(connectClusterId);
+
+        List<ConnectorMetrics> metricsList = new ArrayList<>();
+        for (String connectorName : connectorList.getData()) {
+            ConnectorMetrics metrics = new ConnectorMetrics(connectClusterId, connectorName);
+            metrics.setClusterPhyId(clusterPhyId);
+
+            metricsList.add(metrics);
+            future.runnableTask(
+                    String.format("class=ConnectConnectorMetricCollector||connectClusterId=%d||connectorName=%s", connectClusterId, connectorName),
+                    30000,
+                    () -> collectMetrics(connectClusterId, connectorName, metrics, items)
+            );
+        }
+        future.waitResult(30000);
+
+        this.publishMetric(new ConnectorMetricEvent(this, metricsList));
+
+        return metricsList;
+    }
+
+    @Override
+    public VersionItemTypeEnum collectorType() {
+        return METRIC_CONNECT_CONNECTOR;
+    }
+
+    /**************************************************** private method ****************************************************/
+
+    private void collectMetrics(Long connectClusterId, String connectorName, ConnectorMetrics metrics, List<VersionControlItem> items) {
+        long startTime = System.currentTimeMillis();
+        ConnectorTypeEnum connectorType = connectorService.getConnectorType(connectClusterId, connectorName);
+
+        metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
+
+        for (VersionControlItem v : items) {
+            try {
+                //过滤已测得指标
+                if (metrics.getMetrics().get(v.getName()) != null) {
+                    continue;
+                }
+
+                Result<ConnectorMetrics> ret = connectorMetricService.collectConnectClusterMetricsFromKafka(connectClusterId, connectorName, v.getName(), connectorType);
+                if (null == ret || ret.failed() || null == ret.getData()) {
+                    continue;
+                }
+
+                metrics.putMetric(ret.getData().getMetrics());
+            } catch (Exception e) {
+                LOGGER.error(
+                        "method=collectMetrics||connectClusterId={}||connectorName={}||metric={}||errMsg=exception!",
+                        connectClusterId, connectorName, v.getName(), e
+                );
+            }
+        }
+
+        // 记录采集性能
+        metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, (System.currentTimeMillis() - startTime) / 1000.0f);
+    }
+}

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/metric/connect/mm2/MirrorMakerMetricCollector.java

@@ -0,0 +1,117 @@
+package com.xiaojukeji.know.streaming.km.collector.metric.connect.mm2;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.collector.metric.connect.AbstractConnectMetricCollector;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.ConnectCluster;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.mm2.MirrorMakerTopic;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.mm2.MirrorMakerMetrics;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionControlItem;
+import com.xiaojukeji.know.streaming.km.common.bean.event.metric.mm2.MirrorMakerMetricEvent;
+import com.xiaojukeji.know.streaming.km.common.bean.po.connect.ConnectorPO;
+import com.xiaojukeji.know.streaming.km.common.constant.Constant;
+import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
+import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
+import com.xiaojukeji.know.streaming.km.core.service.connect.connector.ConnectorService;
+import com.xiaojukeji.know.streaming.km.core.service.connect.mm2.MirrorMakerMetricService;
+import com.xiaojukeji.know.streaming.km.core.service.connect.mm2.MirrorMakerService;
+import com.xiaojukeji.know.streaming.km.core.service.version.VersionControlService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.xiaojukeji.know.streaming.km.common.constant.connect.KafkaConnectConstant.MIRROR_MAKER_SOURCE_CONNECTOR_TYPE;
+import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum.METRIC_CONNECT_MIRROR_MAKER;
+
+/**
+ * @author wyb
+ * @date 2022/12/15
+ */
+@Component
+public class MirrorMakerMetricCollector extends AbstractConnectMetricCollector<MirrorMakerMetrics> {
+    protected static final ILog LOGGER = LogFactory.getLog(MirrorMakerMetricCollector.class);
+
+    @Autowired
+    private VersionControlService versionControlService;
+
+    @Autowired
+    private MirrorMakerService mirrorMakerService;
+
+    @Autowired
+    private ConnectorService connectorService;
+
+    @Autowired
+    private MirrorMakerMetricService mirrorMakerMetricService;
+
+    @Override
+    public VersionItemTypeEnum collectorType() {
+        return METRIC_CONNECT_MIRROR_MAKER;
+    }
+
+    @Override
+    public List<MirrorMakerMetrics> collectConnectMetrics(ConnectCluster connectCluster) {
+        Long clusterPhyId     = connectCluster.getKafkaClusterPhyId();
+        Long connectClusterId = connectCluster.getId();
+
+        List<ConnectorPO> mirrorMakerList = connectorService.listByConnectClusterIdFromDB(connectClusterId).stream().filter(elem -> elem.getConnectorClassName().equals(MIRROR_MAKER_SOURCE_CONNECTOR_TYPE)).collect(Collectors.toList());
+        Map<String, MirrorMakerTopic> mirrorMakerTopicMap = mirrorMakerService.getMirrorMakerTopicMap(connectClusterId).getData();
+
+        List<VersionControlItem> items  = versionControlService.listVersionControlItem(this.getClusterVersion(connectCluster), collectorType().getCode());
+        FutureWaitUtil<Void> future     = this.getFutureUtilByClusterPhyId(clusterPhyId);
+
+        List<MirrorMakerMetrics> metricsList = new ArrayList<>();
+
+        for (ConnectorPO mirrorMaker : mirrorMakerList) {
+            MirrorMakerMetrics metrics = new MirrorMakerMetrics(clusterPhyId, connectClusterId, mirrorMaker.getConnectorName());
+            metricsList.add(metrics);
+
+            List<MirrorMakerTopic> mirrorMakerTopicList = mirrorMakerService.getMirrorMakerTopicList(mirrorMaker, mirrorMakerTopicMap);
+            future.runnableTask(String.format("class=MirrorMakerMetricCollector||connectClusterId=%d||mirrorMakerName=%s", connectClusterId, mirrorMaker.getConnectorName()),
+                    30000,
+                    () -> collectMetrics(connectClusterId, mirrorMaker.getConnectorName(), metrics, items, mirrorMakerTopicList));
+        }
+        future.waitResult(30000);
+
+        this.publishMetric(new MirrorMakerMetricEvent(this,metricsList));
+
+        return metricsList;
+    }
+
+    /**************************************************** private method ****************************************************/
+    private void collectMetrics(Long connectClusterId, String mirrorMakerName, MirrorMakerMetrics metrics, List<VersionControlItem> items, List<MirrorMakerTopic> mirrorMakerTopicList) {
+        long startTime = System.currentTimeMillis();
+        metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
+
+        for (VersionControlItem v : items) {
+            try {
+                //已测量指标过滤
+                if (metrics.getMetrics().get(v.getName()) != null) {
+                    continue;
+                }
+
+                Result<MirrorMakerMetrics> ret = mirrorMakerMetricService.collectMirrorMakerMetricsFromKafka(connectClusterId, mirrorMakerName, mirrorMakerTopicList, v.getName());
+                if (ret == null || !ret.hasData()) {
+                    continue;
+                }
+                metrics.putMetric(ret.getData().getMetrics());
+
+            } catch (Exception e) {
+                LOGGER.error(
+                        "method=collectMetrics||connectClusterId={}||mirrorMakerName={}||metric={}||errMsg=exception!",
+                        connectClusterId, mirrorMakerName, v.getName(), e
+                );
+
+            }
+        }
+        metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, (System.currentTimeMillis() - startTime) / 1000.0f);
+
+    }
+}
+
+
+

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/metric/kafka/AbstractKafkaMetricCollector.java

@@ -0,0 +1,50 @@
+package com.xiaojukeji.know.streaming.km.collector.metric.kafka;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.collector.metric.AbstractMetricCollector;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
+import com.xiaojukeji.know.streaming.km.common.utils.LoggerUtil;
+import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterPhyService;
+import org.springframework.beans.factory.annotation.Autowired;
+
+import java.util.List;
+
+/**
+ * @author didi
+ */
+public abstract class AbstractKafkaMetricCollector<M> extends AbstractMetricCollector<M, ClusterPhy> {
+    private static final ILog LOGGER                    = LogFactory.getLog(AbstractMetricCollector.class);
+
+    protected static final ILog METRIC_COLLECTED_LOGGER = LoggerUtil.getMetricCollectedLogger();
+
+    @Autowired
+    private ClusterPhyService clusterPhyService;
+
+    public abstract List<M> collectKafkaMetrics(ClusterPhy clusterPhy);
+
+    @Override
+    public String getClusterVersion(ClusterPhy clusterPhy){
+        return clusterPhyService.getVersionFromCacheFirst(clusterPhy.getId());
+    }
+
+    @Override
+    public void collectMetrics(ClusterPhy clusterPhy) {
+        long startTime = System.currentTimeMillis();
+
+        // 采集指标
+        List<M> metricsList = this.collectKafkaMetrics(clusterPhy);
+
+        // 输出耗时信息
+        LOGGER.info(
+                "metricType={}||clusterPhyId={}||costTimeUnitMs={}",
+                this.collectorType().getMessage(), clusterPhy.getId(), System.currentTimeMillis() - startTime
+        );
+
+        // 输出采集到的指标信息
+        METRIC_COLLECTED_LOGGER.debug("metricType={}||clusterPhyId={}||metrics={}!",
+                this.collectorType().getMessage(), clusterPhy.getId(), ConvertUtil.obj2Json(metricsList)
+        );
+    }
+}

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/metric/kafka/BrokerMetricCollector.java

@@ -1,6 +1,5 @@
-package com.xiaojukeji.know.streaming.km.collector.metric;
+package com.xiaojukeji.know.streaming.km.collector.metric.kafka;
 
-import com.alibaba.fastjson.JSON;
 import com.didiglobal.logi.log.ILog;
 import com.didiglobal.logi.log.LogFactory;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.broker.Broker;
@@ -11,7 +10,6 @@ import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionContro
 import com.xiaojukeji.know.streaming.km.common.bean.event.metric.BrokerMetricEvent;
 import com.xiaojukeji.know.streaming.km.common.constant.Constant;
 import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
-import com.xiaojukeji.know.streaming.km.common.utils.EnvUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
 import com.xiaojukeji.know.streaming.km.core.service.broker.BrokerMetricService;
 import com.xiaojukeji.know.streaming.km.core.service.broker.BrokerService;
@@ -28,8 +26,8 @@ import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemT
  * @author didi
  */
 @Component
-public class BrokerMetricCollector extends AbstractMetricCollector<BrokerMetrics> {
+public class BrokerMetricCollector extends AbstractKafkaMetricCollector<BrokerMetrics> {
-    protected static final ILog LOGGER = LogFactory.getLog("METRIC_LOGGER");
+    private static final ILog LOGGER = LogFactory.getLog(BrokerMetricCollector.class);
 
     @Autowired
     private VersionControlService versionControlService;
@@ -41,32 +39,31 @@ public class BrokerMetricCollector extends AbstractMetricCollector<BrokerMetrics
     private BrokerService brokerService;
 
     @Override
-    public void collectMetrics(ClusterPhy clusterPhy) {
+    public List<BrokerMetrics> collectKafkaMetrics(ClusterPhy clusterPhy) {
-        Long startTime     = System.currentTimeMillis();
         Long clusterPhyId  = clusterPhy.getId();
 
         List<Broker>             brokers = brokerService.listAliveBrokersFromDB(clusterPhy.getId());
-        List<VersionControlItem> items   = versionControlService.listVersionControlItem(clusterPhyId, collectorType().getCode());
+        List<VersionControlItem> items   = versionControlService.listVersionControlItem(this.getClusterVersion(clusterPhy), collectorType().getCode());
 
         FutureWaitUtil<Void> future = this.getFutureUtilByClusterPhyId(clusterPhyId);
 
-        List<BrokerMetrics> brokerMetrics = new ArrayList<>();
+        List<BrokerMetrics> metricsList = new ArrayList<>();
         for(Broker broker : brokers) {
             BrokerMetrics metrics = new BrokerMetrics(clusterPhyId, broker.getBrokerId(), broker.getHost(), broker.getPort());
-            brokerMetrics.add(metrics);
+            metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
+            metricsList.add(metrics);
 
             future.runnableTask(
-                    String.format("method=BrokerMetricCollector||clusterPhyId=%d||brokerId=%d", clusterPhyId, broker.getBrokerId()),
+                    String.format("class=BrokerMetricCollector||clusterPhyId=%d||brokerId=%d", clusterPhyId, broker.getBrokerId()),
                     30000,
                     () -> collectMetrics(clusterPhyId, metrics, items)
             );
         }
 
         future.waitExecute(30000);
-        this.publishMetric(new BrokerMetricEvent(this, brokerMetrics));
+        this.publishMetric(new BrokerMetricEvent(this, metricsList));
 
-        LOGGER.info("method=BrokerMetricCollector||clusterPhyId={}||startTime={}||costTime={}||msg=collect finished.",
+        return metricsList;
-                clusterPhyId, startTime, System.currentTimeMillis() - startTime);
     }
 
     @Override
@@ -78,7 +75,6 @@ public class BrokerMetricCollector extends AbstractMetricCollector<BrokerMetrics
 
     private void collectMetrics(Long clusterPhyId, BrokerMetrics metrics, List<VersionControlItem> items) {
         long startTime = System.currentTimeMillis();
-        metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
 
         for(VersionControlItem v : items) {
             try {
@@ -92,14 +88,11 @@ public class BrokerMetricCollector extends AbstractMetricCollector<BrokerMetrics
                 }
 
                 metrics.putMetric(ret.getData().getMetrics());
-
-                if(!EnvUtil.isOnline()){
-                    LOGGER.info("method=BrokerMetricCollector||clusterId={}||brokerId={}||metric={}||metric={}!",
-                            clusterPhyId, metrics.getBrokerId(), v.getName(), JSON.toJSONString(ret.getData().getMetrics()));
-                }
             } catch (Exception e){
-                LOGGER.error("method=BrokerMetricCollector||clusterId={}||brokerId={}||metric={}||errMsg=exception!",
+                LOGGER.error(
-                        clusterPhyId, metrics.getBrokerId(), v.getName(), e);
+                        "method=collectMetrics||clusterPhyId={}||brokerId={}||metricName={}||errMsg=exception!",
+                        clusterPhyId, metrics.getBrokerId(), v.getName(), e
+                );
             }
         }
 

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/metric/kafka/ClusterMetricCollector.java

@@ -1,4 +1,4 @@
-package com.xiaojukeji.know.streaming.km.collector.metric;
+package com.xiaojukeji.know.streaming.km.collector.metric.kafka;
 
 import com.didiglobal.logi.log.ILog;
 import com.didiglobal.logi.log.LogFactory;
@@ -7,18 +7,15 @@ import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.ClusterMetric
 import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionControlItem;
 import com.xiaojukeji.know.streaming.km.common.bean.event.metric.ClusterMetricEvent;
-import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.ClusterMetricPO;
 import com.xiaojukeji.know.streaming.km.common.constant.Constant;
 import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
-import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
-import com.xiaojukeji.know.streaming.km.common.utils.EnvUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
 import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterMetricService;
 import com.xiaojukeji.know.streaming.km.core.service.version.VersionControlService;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Component;
 
-import java.util.Arrays;
+import java.util.Collections;
 import java.util.List;
 
 import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum.METRIC_CLUSTER;
@@ -27,8 +24,8 @@ import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemT
  * @author didi
  */
 @Component
-public class ClusterMetricCollector extends AbstractMetricCollector<ClusterMetricPO> {
+public class ClusterMetricCollector extends AbstractKafkaMetricCollector<ClusterMetrics> {
-    protected static final ILog  LOGGER = LogFactory.getLog("METRIC_LOGGER");
+    protected static final ILog LOGGER = LogFactory.getLog(ClusterMetricCollector.class);
 
     @Autowired
     private VersionControlService versionControlService;
@@ -37,35 +34,37 @@ public class ClusterMetricCollector extends AbstractMetricCollector<ClusterMetri
     private ClusterMetricService  clusterMetricService;
 
     @Override
-    public void collectMetrics(ClusterPhy clusterPhy) {
+    public List<ClusterMetrics> collectKafkaMetrics(ClusterPhy clusterPhy) {
         Long        startTime           =   System.currentTimeMillis();
         Long        clusterPhyId        =   clusterPhy.getId();
-        List<VersionControlItem> items  =   versionControlService.listVersionControlItem(clusterPhyId, collectorType().getCode());
+        List<VersionControlItem> items  =   versionControlService.listVersionControlItem(this.getClusterVersion(clusterPhy), collectorType().getCode());
 
         ClusterMetrics metrics = new ClusterMetrics(clusterPhyId, clusterPhy.getKafkaVersion());
+        metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
 
         FutureWaitUtil<Void> future = this.getFutureUtilByClusterPhyId(clusterPhyId);
 
         for(VersionControlItem v : items) {
             future.runnableTask(
-                    String.format("method=ClusterMetricCollector||clusterPhyId=%d||metricName=%s", clusterPhyId, v.getName()),
+                    String.format("class=ClusterMetricCollector||clusterPhyId=%d||metricName=%s", clusterPhyId, v.getName()),
                     30000,
                     () -> {
                         try {
-                            if(null != metrics.getMetrics().get(v.getName())){return null;}
+                            if(null != metrics.getMetrics().get(v.getName())){
+                                return null;
+                            }
 
                             Result<ClusterMetrics> ret = clusterMetricService.collectClusterMetricsFromKafka(clusterPhyId, v.getName());
-                            if(null == ret || ret.failed() || null == ret.getData()){return null;}
+                            if(null == ret || ret.failed() || null == ret.getData()){
+                                return null;
+                            }
 
                             metrics.putMetric(ret.getData().getMetrics());
-
-                            if(!EnvUtil.isOnline()){
-                                LOGGER.info("method=ClusterMetricCollector||clusterPhyId={}||metricName={}||metricValue={}",
-                                        clusterPhyId, v.getName(), ConvertUtil.obj2Json(ret.getData().getMetrics()));
-                            }
                         } catch (Exception e){
-                            LOGGER.error("method=ClusterMetricCollector||clusterPhyId={}||metricName={}||errMsg=exception!",
+                            LOGGER.error(
-                                    clusterPhyId, v.getName(), e);
+                                    "method=collectKafkaMetrics||clusterPhyId={}||metricName={}||errMsg=exception!",
+                                    clusterPhyId, v.getName(), e
+                            );
                         }
 
                         return null;
@@ -76,10 +75,9 @@ public class ClusterMetricCollector extends AbstractMetricCollector<ClusterMetri
 
         metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, (System.currentTimeMillis() - startTime) / 1000.0f);
 
-        publishMetric(new ClusterMetricEvent(this, Arrays.asList(metrics)));
+        publishMetric(new ClusterMetricEvent(this, Collections.singletonList(metrics)));
 
-        LOGGER.info("method=ClusterMetricCollector||clusterPhyId={}||startTime={}||costTime={}||msg=msg=collect finished.",
+        return Collections.singletonList(metrics);
-                clusterPhyId, startTime, System.currentTimeMillis() - startTime);
     }
 
     @Override

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/metric/kafka/GroupMetricCollector.java

@@ -1,6 +1,5 @@
-package com.xiaojukeji.know.streaming.km.collector.metric;
+package com.xiaojukeji.know.streaming.km.collector.metric.kafka;
 
-import com.alibaba.fastjson.JSON;
 import com.didiglobal.logi.log.ILog;
 import com.didiglobal.logi.log.LogFactory;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
@@ -10,20 +9,16 @@ import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionContro
 import com.xiaojukeji.know.streaming.km.common.bean.event.metric.GroupMetricEvent;
 import com.xiaojukeji.know.streaming.km.common.constant.Constant;
 import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
-import com.xiaojukeji.know.streaming.km.common.utils.EnvUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
 import com.xiaojukeji.know.streaming.km.core.service.group.GroupMetricService;
 import com.xiaojukeji.know.streaming.km.core.service.group.GroupService;
 import com.xiaojukeji.know.streaming.km.core.service.version.VersionControlService;
-import org.apache.commons.collections.CollectionUtils;
+import org.apache.kafka.common.TopicPartition;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Component;
 
-import java.util.ArrayList;
+import java.util.*;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
 import java.util.concurrent.ConcurrentHashMap;
 
 import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum.METRIC_GROUP;
@@ -32,8 +27,8 @@ import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemT
  * @author didi
  */
 @Component
-public class GroupMetricCollector extends AbstractMetricCollector<List<GroupMetrics>> {
+public class GroupMetricCollector extends AbstractKafkaMetricCollector<GroupMetrics> {
-    protected static final ILog  LOGGER = LogFactory.getLog("METRIC_LOGGER");
+    protected static final ILog  LOGGER = LogFactory.getLog(GroupMetricCollector.class);
 
     @Autowired
     private VersionControlService versionControlService;
@@ -45,40 +40,38 @@ public class GroupMetricCollector extends AbstractMetricCollector<List<GroupMetr
     private GroupService groupService;
 
     @Override
-    public void collectMetrics(ClusterPhy clusterPhy) {
+    public List<GroupMetrics> collectKafkaMetrics(ClusterPhy clusterPhy) {
-        Long   startTime        = System.currentTimeMillis();
         Long   clusterPhyId     = clusterPhy.getId();
 
-        List<String> groups = new ArrayList<>();
+        List<String> groupNameList = new ArrayList<>();
         try {
-            groups = groupService.listGroupsFromKafka(clusterPhyId);
+            groupNameList = groupService.listGroupsFromKafka(clusterPhy);
         } catch (Exception e) {
-            LOGGER.error("method=GroupMetricCollector||clusterPhyId={}||msg=exception!", clusterPhyId, e);
+            LOGGER.error("method=collectKafkaMetrics||clusterPhyId={}||msg=exception!", clusterPhyId, e);
         }
 
-        if(CollectionUtils.isEmpty(groups)){return;}
+        if(ValidateUtils.isEmptyList(groupNameList)) {
+            return Collections.emptyList();
+        }
 
-        List<VersionControlItem> items = versionControlService.listVersionControlItem(clusterPhyId, collectorType().getCode());
+        List<VersionControlItem> items = versionControlService.listVersionControlItem(this.getClusterVersion(clusterPhy), collectorType().getCode());
 
-        FutureWaitUtil<Void> future = getFutureUtilByClusterPhyId(clusterPhyId);
+        FutureWaitUtil<Void> future = this.getFutureUtilByClusterPhyId(clusterPhyId);
 
         Map<String, List<GroupMetrics>> metricsMap = new ConcurrentHashMap<>();
-        for(String groupName : groups) {
+        for(String groupName : groupNameList) {
             future.runnableTask(
-                    String.format("method=GroupMetricCollector||clusterPhyId=%d||groupName=%s", clusterPhyId, groupName),
+                    String.format("class=GroupMetricCollector||clusterPhyId=%d||groupName=%s", clusterPhyId, groupName),
                     30000,
                     () -> collectMetrics(clusterPhyId, groupName, metricsMap, items));
         }
 
         future.waitResult(30000);
 
-        List<GroupMetrics> metricsList = new ArrayList<>();
+        List<GroupMetrics> metricsList = metricsMap.values().stream().collect(ArrayList::new, ArrayList::addAll, ArrayList::addAll);
-        metricsMap.values().forEach(elem -> metricsList.addAll(elem));
 
         publishMetric(new GroupMetricEvent(this, metricsList));
-
+        return metricsList;
-        LOGGER.info("method=GroupMetricCollector||clusterPhyId={}||startTime={}||cost={}||msg=collect finished.",
-                clusterPhyId, startTime, System.currentTimeMillis() - startTime);
     }
 
     @Override
@@ -91,9 +84,7 @@ public class GroupMetricCollector extends AbstractMetricCollector<List<GroupMetr
     private void collectMetrics(Long clusterPhyId, String groupName, Map<String, List<GroupMetrics>> metricsMap, List<VersionControlItem> items) {
         long startTime = System.currentTimeMillis();
 
-        List<GroupMetrics> groupMetricsList = new ArrayList<>();
+        Map<TopicPartition, GroupMetrics> subMetricMap = new HashMap<>();
-
-        Map<String, GroupMetrics> tpGroupPOMap = new HashMap<>();
 
         GroupMetrics groupMetrics = new GroupMetrics(clusterPhyId, groupName, true);
         groupMetrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
@@ -107,38 +98,31 @@ public class GroupMetricCollector extends AbstractMetricCollector<List<GroupMetr
                     continue;
                 }
 
-                ret.getData().stream().forEach(metrics -> {
+                ret.getData().forEach(metrics -> {
                     if (metrics.isBGroupMetric()) {
                         groupMetrics.putMetric(metrics.getMetrics());
-                    } else {
+                        return;
-                        String  topicName   = metrics.getTopic();
-                        Integer partitionId = metrics.getPartitionId();
-                        String  tpGroupKey  = genTopicPartitionGroupKey(topicName, partitionId);
-
-                        tpGroupPOMap.putIfAbsent(tpGroupKey, new GroupMetrics(clusterPhyId, partitionId, topicName, groupName, false));
-                        tpGroupPOMap.get(tpGroupKey).putMetric(metrics.getMetrics());
                     }
-                });
 
-                if(!EnvUtil.isOnline()){
+                    TopicPartition tp  = new TopicPartition(metrics.getTopic(), metrics.getPartitionId());
-                    LOGGER.info("method=GroupMetricCollector||clusterPhyId={}||groupName={}||metricName={}||metricValue={}",
+                    subMetricMap.putIfAbsent(tp, new GroupMetrics(clusterPhyId, metrics.getPartitionId(), metrics.getTopic(), groupName, false));
-                            clusterPhyId, groupName, metricName, JSON.toJSONString(ret.getData()));
+                    subMetricMap.get(tp).putMetric(metrics.getMetrics());
-                }
+                });
-            }catch (Exception e){
+            } catch (Exception e) {
-                LOGGER.error("method=GroupMetricCollector||clusterPhyId={}||groupName={}||errMsg=exception!", clusterPhyId, groupName, e);
+                LOGGER.error(
+                        "method=collectMetrics||clusterPhyId={}||groupName={}||errMsg=exception!",
+                        clusterPhyId, groupName, e
+                );
             }
         }
 
-        groupMetricsList.add(groupMetrics);
+        List<GroupMetrics> metricsList = new ArrayList<>();
-        groupMetricsList.addAll(tpGroupPOMap.values());
+        metricsList.add(groupMetrics);
+        metricsList.addAll(subMetricMap.values());
 
         // 记录采集性能
         groupMetrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, (System.currentTimeMillis() - startTime) / 1000.0f);
 
-        metricsMap.put(groupName, groupMetricsList);
+        metricsMap.put(groupName, metricsList);
-    }
-
-    private String genTopicPartitionGroupKey(String topic, Integer partitionId){
-        return topic + "@" + partitionId;
     }
 }

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/metric/kafka/PartitionMetricCollector.java

@@ -1,4 +1,4 @@
-package com.xiaojukeji.know.streaming.km.collector.metric;
+package com.xiaojukeji.know.streaming.km.collector.metric.kafka;
 
 import com.didiglobal.logi.log.ILog;
 import com.didiglobal.logi.log.LogFactory;
@@ -9,8 +9,6 @@ import com.xiaojukeji.know.streaming.km.common.bean.entity.topic.Topic;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionControlItem;
 import com.xiaojukeji.know.streaming.km.common.bean.event.metric.PartitionMetricEvent;
 import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
-import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
-import com.xiaojukeji.know.streaming.km.common.utils.EnvUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
 import com.xiaojukeji.know.streaming.km.core.service.partition.PartitionMetricService;
 import com.xiaojukeji.know.streaming.km.core.service.topic.TopicService;
@@ -27,8 +25,8 @@ import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemT
  * @author didi
  */
 @Component
-public class PartitionMetricCollector extends AbstractMetricCollector<PartitionMetrics> {
+public class PartitionMetricCollector extends AbstractKafkaMetricCollector<PartitionMetrics> {
-    protected static final ILog  LOGGER = LogFactory.getLog("METRIC_LOGGER");
+    protected static final ILog  LOGGER = LogFactory.getLog(PartitionMetricCollector.class);
 
     @Autowired
     private VersionControlService versionControlService;
@@ -40,13 +38,10 @@ public class PartitionMetricCollector extends AbstractMetricCollector<PartitionM
     private TopicService topicService;
 
     @Override
-    public void collectMetrics(ClusterPhy clusterPhy) {
+    public List<PartitionMetrics> collectKafkaMetrics(ClusterPhy clusterPhy) {
-        Long        startTime           =   System.currentTimeMillis();
         Long        clusterPhyId        =   clusterPhy.getId();
         List<Topic> topicList           =   topicService.listTopicsFromCacheFirst(clusterPhyId);
-        List<VersionControlItem> items  =   versionControlService.listVersionControlItem(clusterPhyId, collectorType().getCode());
+        List<VersionControlItem> items  =   versionControlService.listVersionControlItem(this.getClusterVersion(clusterPhy), collectorType().getCode());
-
-        // 获取集群所有分区
 
         FutureWaitUtil<Void> future = this.getFutureUtilByClusterPhyId(clusterPhyId);
 
@@ -55,9 +50,9 @@ public class PartitionMetricCollector extends AbstractMetricCollector<PartitionM
             metricsMap.put(topic.getTopicName(), new ConcurrentHashMap<>());
 
             future.runnableTask(
-                    String.format("method=PartitionMetricCollector||clusterPhyId=%d||topicName=%s", clusterPhyId, topic.getTopicName()),
+                    String.format("class=PartitionMetricCollector||clusterPhyId=%d||topicName=%s", clusterPhyId, topic.getTopicName()),
                     30000,
-                    () -> collectMetrics(clusterPhyId, topic.getTopicName(), metricsMap.get(topic.getTopicName()), items)
+                    () -> this.collectMetrics(clusterPhyId, topic.getTopicName(), metricsMap.get(topic.getTopicName()), items)
             );
         }
 
@@ -68,10 +63,7 @@ public class PartitionMetricCollector extends AbstractMetricCollector<PartitionM
 
         this.publishMetric(new PartitionMetricEvent(this, metricsList));
 
-        LOGGER.info(
+        return metricsList;
-                "method=PartitionMetricCollector||clusterPhyId={}||startTime={}||costTime={}||msg=collect finished.",
-                clusterPhyId, startTime, System.currentTimeMillis() - startTime
-        );
     }
 
     @Override
@@ -109,17 +101,9 @@ public class PartitionMetricCollector extends AbstractMetricCollector<PartitionM
                     PartitionMetrics allMetrics = metricsMap.get(subMetrics.getPartitionId());
                     allMetrics.putMetric(subMetrics.getMetrics());
                 }
-
-                if (!EnvUtil.isOnline()) {
-                    LOGGER.info(
-                            "class=PartitionMetricCollector||method=collectMetrics||clusterPhyId={}||topicName={}||metricName={}||metricValue={}!",
-                            clusterPhyId, topicName, v.getName(), ConvertUtil.obj2Json(ret.getData())
-                    );
-                }
-
             } catch (Exception e) {
                 LOGGER.info(
-                        "class=PartitionMetricCollector||method=collectMetrics||clusterPhyId={}||topicName={}||metricName={}||errMsg=exception",
+                        "method=collectMetrics||clusterPhyId={}||topicName={}||metricName={}||errMsg=exception",
                         clusterPhyId, topicName, v.getName(), e
                 );
             }

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/metric/kafka/TopicMetricCollector.java

@@ -1,4 +1,4 @@
-package com.xiaojukeji.know.streaming.km.collector.metric;
+package com.xiaojukeji.know.streaming.km.collector.metric.kafka;
 
 import com.didiglobal.logi.log.ILog;
 import com.didiglobal.logi.log.LogFactory;
@@ -10,8 +10,6 @@ import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionContro
 import com.xiaojukeji.know.streaming.km.common.bean.event.metric.TopicMetricEvent;
 import com.xiaojukeji.know.streaming.km.common.constant.Constant;
 import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
-import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
-import com.xiaojukeji.know.streaming.km.common.utils.EnvUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
 import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
 import com.xiaojukeji.know.streaming.km.core.service.topic.TopicMetricService;
@@ -31,8 +29,8 @@ import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemT
  * @author didi
  */
 @Component
-public class TopicMetricCollector extends AbstractMetricCollector<List<TopicMetrics>> {
+public class TopicMetricCollector extends AbstractKafkaMetricCollector<TopicMetrics> {
-    protected static final ILog  LOGGER = LogFactory.getLog("METRIC_LOGGER");
+    protected static final ILog  LOGGER = LogFactory.getLog(TopicMetricCollector.class);
 
     @Autowired
     private VersionControlService versionControlService;
@@ -46,11 +44,10 @@ public class TopicMetricCollector extends AbstractMetricCollector<List<TopicMetr
     private static final Integer AGG_METRICS_BROKER_ID = -10000;
 
     @Override
-    public void collectMetrics(ClusterPhy clusterPhy) {
+    public List<TopicMetrics> collectKafkaMetrics(ClusterPhy clusterPhy) {
-        Long        startTime           =   System.currentTimeMillis();
         Long        clusterPhyId        =   clusterPhy.getId();
         List<Topic> topics              =   topicService.listTopicsFromCacheFirst(clusterPhyId);
-        List<VersionControlItem> items  =   versionControlService.listVersionControlItem(clusterPhyId, collectorType().getCode());
+        List<VersionControlItem> items  =   versionControlService.listVersionControlItem(this.getClusterVersion(clusterPhy), collectorType().getCode());
 
         FutureWaitUtil<Void> future = this.getFutureUtilByClusterPhyId(clusterPhyId);
 
@@ -64,7 +61,7 @@ public class TopicMetricCollector extends AbstractMetricCollector<List<TopicMetr
             allMetricsMap.put(topic.getTopicName(), metricsMap);
 
             future.runnableTask(
-                    String.format("method=TopicMetricCollector||clusterPhyId=%d||topicName=%s", clusterPhyId, topic.getTopicName()),
+                    String.format("class=TopicMetricCollector||clusterPhyId=%d||topicName=%s", clusterPhyId, topic.getTopicName()),
                     30000,
                     () -> collectMetrics(clusterPhyId, topic.getTopicName(), metricsMap, items)
             );
@@ -77,8 +74,7 @@ public class TopicMetricCollector extends AbstractMetricCollector<List<TopicMetr
 
         this.publishMetric(new TopicMetricEvent(this, metricsList));
 
-        LOGGER.info("method=TopicMetricCollector||clusterPhyId={}||startTime={}||costTime={}||msg=collect finished.",
+        return metricsList;
-                clusterPhyId, startTime, System.currentTimeMillis() - startTime);
     }
 
     @Override
@@ -118,14 +114,9 @@ public class TopicMetricCollector extends AbstractMetricCollector<List<TopicMetr
                         metricsMap.get(metrics.getBrokerId()).putMetric(metrics.getMetrics());
                     }
                 });
-
-                if (!EnvUtil.isOnline()) {
-                    LOGGER.info("method=TopicMetricCollector||clusterPhyId={}||topicName={}||metricName={}||metricValue={}.",
-                            clusterPhyId, topicName, v.getName(), ConvertUtil.obj2Json(ret.getData())
-                    );
-                }
             } catch (Exception e) {
-                LOGGER.error("method=TopicMetricCollector||clusterPhyId={}||topicName={}||metricName={}||errMsg=exception!",
+                LOGGER.error(
+                        "method=collectMetrics||clusterPhyId={}||topicName={}||metricName={}||errMsg=exception!",
                         clusterPhyId, topicName, v.getName(), e
                 );
             }

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/metric/kafka/ZookeeperMetricCollector.java

@@ -0,0 +1,111 @@
+package com.xiaojukeji.know.streaming.km.collector.metric.kafka;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.config.ZKConfig;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.kafkacontroller.KafkaController;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.param.metric.ZookeeperMetricParam;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionControlItem;
+import com.xiaojukeji.know.streaming.km.common.utils.Tuple;
+import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
+import com.xiaojukeji.know.streaming.km.common.bean.event.metric.ZookeeperMetricEvent;
+import com.xiaojukeji.know.streaming.km.common.constant.Constant;
+import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.zookeeper.ZookeeperInfo;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.ZookeeperMetrics;
+import com.xiaojukeji.know.streaming.km.core.service.kafkacontroller.KafkaControllerService;
+import com.xiaojukeji.know.streaming.km.core.service.version.VersionControlService;
+import com.xiaojukeji.know.streaming.km.core.service.zookeeper.ZookeeperMetricService;
+import com.xiaojukeji.know.streaming.km.core.service.zookeeper.ZookeeperService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum.METRIC_ZOOKEEPER;
+
+/**
+ * @author didi
+ */
+@Component
+public class ZookeeperMetricCollector extends AbstractKafkaMetricCollector<ZookeeperMetrics> {
+    protected static final ILog  LOGGER = LogFactory.getLog(ZookeeperMetricCollector.class);
+
+    @Autowired
+    private VersionControlService versionControlService;
+
+    @Autowired
+    private ZookeeperMetricService zookeeperMetricService;
+
+    @Autowired
+    private ZookeeperService zookeeperService;
+
+    @Autowired
+    private KafkaControllerService kafkaControllerService;
+
+    @Override
+    public List<ZookeeperMetrics> collectKafkaMetrics(ClusterPhy clusterPhy) {
+        Long        startTime           =   System.currentTimeMillis();
+        Long        clusterPhyId        =   clusterPhy.getId();
+        List<VersionControlItem> items  =   versionControlService.listVersionControlItem(this.getClusterVersion(clusterPhy), collectorType().getCode());
+        List<ZookeeperInfo> aliveZKList =   zookeeperService.listFromDBByCluster(clusterPhyId)
+                .stream()
+                .filter(elem -> Constant.ALIVE.equals(elem.getStatus()))
+                .collect(Collectors.toList());
+        KafkaController kafkaController =   kafkaControllerService.getKafkaControllerFromDB(clusterPhyId);
+
+        ZookeeperMetrics metrics = ZookeeperMetrics.initWithMetric(clusterPhyId, Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
+        if (ValidateUtils.isEmptyList(aliveZKList)) {
+            // 没有存活的ZK时，发布事件，然后直接返回
+            publishMetric(new ZookeeperMetricEvent(this, Collections.singletonList(metrics)));
+            return Collections.singletonList(metrics);
+        }
+
+        // 构造参数
+        ZookeeperMetricParam param = new ZookeeperMetricParam(
+                clusterPhyId,
+                aliveZKList.stream().map(elem -> new Tuple<String, Integer>(elem.getHost(), elem.getPort())).collect(Collectors.toList()),
+                ConvertUtil.str2ObjByJson(clusterPhy.getZkProperties(), ZKConfig.class),
+                kafkaController == null? Constant.INVALID_CODE: kafkaController.getBrokerId(),
+                null
+        );
+
+        for(VersionControlItem v : items) {
+            try {
+                if(null != metrics.getMetrics().get(v.getName())) {
+                    continue;
+                }
+
+                param.setMetricName(v.getName());
+
+                Result<ZookeeperMetrics> ret = zookeeperMetricService.collectMetricsFromZookeeper(param);
+                if(null == ret || ret.failed() || null == ret.getData()){
+                    continue;
+                }
+
+                metrics.putMetric(ret.getData().getMetrics());
+            } catch (Exception e){
+                LOGGER.error(
+                        "method=collectMetrics||clusterPhyId={}||metricName={}||errMsg=exception!",
+                        clusterPhyId, v.getName(), e
+                );
+            }
+        }
+
+        metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, (System.currentTimeMillis() - startTime) / 1000.0f);
+
+        this.publishMetric(new ZookeeperMetricEvent(this, Collections.singletonList(metrics)));
+
+        return Collections.singletonList(metrics);
+    }
+
+    @Override
+    public VersionItemTypeEnum collectorType() {
+        return METRIC_ZOOKEEPER;
+    }
+}

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/service/CollectThreadPoolService.java

@@ -237,7 +237,7 @@ public class CollectThreadPoolService {
     private synchronized FutureWaitUtil<Void> closeOldAndCreateNew(Long shardId) {
         // 新的
         FutureWaitUtil<Void> newFutureUtil = FutureWaitUtil.init(
-                "CollectorMetricsFutureUtil-Shard-" + shardId,
+                "MetricCollect-Shard-" + shardId,
                 this.futureUtilThreadNum,
                 this.futureUtilThreadNum,
                 this.futureUtilQueueSize

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/sink/AbstractMetricESSender.java

@@ -0,0 +1,52 @@
+package com.xiaojukeji.know.streaming.km.collector.sink;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.common.bean.po.BaseESPO;
+import com.xiaojukeji.know.streaming.km.common.utils.FutureUtil;
+import com.xiaojukeji.know.streaming.km.persistence.es.dao.BaseMetricESDAO;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.Objects;
+
+public abstract class AbstractMetricESSender {
+    private static final ILog LOGGER = LogFactory.getLog(AbstractMetricESSender.class);
+
+    private static final int THRESHOLD = 100;
+
+    private static final FutureUtil<Void> esExecutor = FutureUtil.init(
+            "MetricsESSender",
+            10,
+            20,
+            10000
+    );
+
+    /**
+     * 根据不同监控维度来发送
+     */
+    protected boolean send2es(String index, List<? extends BaseESPO> statsList) {
+        LOGGER.info("method=send2es||indexName={}||metricsSize={}||msg=send metrics to es", index, statsList.size());
+
+        if (CollectionUtils.isEmpty(statsList)) {
+            return true;
+        }
+
+        BaseMetricESDAO baseMetricESDao = BaseMetricESDAO.getByStatsType(index);
+        if (Objects.isNull(baseMetricESDao)) {
+            LOGGER.error("method=send2es||indexName={}||errMsg=find dao failed", index);
+            return false;
+        }
+
+        for (int i = 0; i < statsList.size(); i += THRESHOLD) {
+            final int idxStart = i;
+
+            // 异步发送
+            esExecutor.submitTask(
+                    () -> baseMetricESDao.batchInsertStats(statsList.subList(idxStart, Math.min(idxStart + THRESHOLD, statsList.size())))
+            );
+        }
+
+        return true;
+    }
+}

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/sink/connect/ConnectClusterMetricESSender.java

@@ -0,0 +1,33 @@
+package com.xiaojukeji.know.streaming.km.collector.sink.connect;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
+import com.xiaojukeji.know.streaming.km.common.bean.event.metric.connect.ConnectClusterMetricEvent;
+import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.connect.ConnectClusterMetricPO;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
+import org.springframework.context.ApplicationListener;
+import org.springframework.stereotype.Component;
+
+import javax.annotation.PostConstruct;
+
+import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.CONNECT_CLUSTER_INDEX;
+
+/**
+ * @author wyb
+ * @date 2022/11/7
+ */
+@Component
+public class ConnectClusterMetricESSender extends AbstractMetricESSender implements ApplicationListener<ConnectClusterMetricEvent> {
+    protected static final ILog LOGGER = LogFactory.getLog(ConnectClusterMetricESSender.class);
+
+    @PostConstruct
+    public void init(){
+        LOGGER.info("class=ConnectClusterMetricESSender||method=init||msg=init finished");
+    }
+
+    @Override
+    public void onApplicationEvent(ConnectClusterMetricEvent event) {
+        send2es(CONNECT_CLUSTER_INDEX, ConvertUtil.list2List(event.getConnectClusterMetrics(), ConnectClusterMetricPO.class));
+    }
+}

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/sink/connect/ConnectorMetricESSender.java

@@ -0,0 +1,33 @@
+package com.xiaojukeji.know.streaming.km.collector.sink.connect;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
+import com.xiaojukeji.know.streaming.km.common.bean.event.metric.connect.ConnectorMetricEvent;
+import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.connect.ConnectorMetricPO;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
+import org.springframework.context.ApplicationListener;
+import org.springframework.stereotype.Component;
+
+import javax.annotation.PostConstruct;
+
+import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.CONNECT_CONNECTOR_INDEX;
+
+/**
+ * @author wyb
+ * @date 2022/11/7
+ */
+@Component
+public class ConnectorMetricESSender extends AbstractMetricESSender implements ApplicationListener<ConnectorMetricEvent> {
+    protected static final ILog LOGGER = LogFactory.getLog(ConnectorMetricESSender.class);
+
+    @PostConstruct
+    public void init(){
+        LOGGER.info("class=ConnectorMetricESSender||method=init||msg=init finished");
+    }
+
+    @Override
+    public void onApplicationEvent(ConnectorMetricEvent event) {
+        send2es(CONNECT_CONNECTOR_INDEX, ConvertUtil.list2List(event.getConnectorMetricsList(), ConnectorMetricPO.class));
+    }
+}

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/sink/kafka/BrokerMetricESSender.java

@@ -0,0 +1,29 @@
+package com.xiaojukeji.know.streaming.km.collector.sink.kafka;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
+import com.xiaojukeji.know.streaming.km.common.bean.event.metric.BrokerMetricEvent;
+import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.BrokerMetricPO;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
+import org.springframework.context.ApplicationListener;
+import org.springframework.stereotype.Component;
+
+import javax.annotation.PostConstruct;
+
+import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.BROKER_INDEX;
+
+@Component
+public class BrokerMetricESSender extends AbstractMetricESSender implements ApplicationListener<BrokerMetricEvent> {
+    private static final ILog  LOGGER = LogFactory.getLog(BrokerMetricESSender.class);
+
+    @PostConstruct
+    public void init(){
+        LOGGER.info("method=init||msg=init finished");
+    }
+
+    @Override
+    public void onApplicationEvent(BrokerMetricEvent event) {
+        send2es(BROKER_INDEX, ConvertUtil.list2List(event.getBrokerMetrics(), BrokerMetricPO.class));
+    }
+}

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/sink/kafka/ClusterMetricESSender.java

@@ -0,0 +1,29 @@
+package com.xiaojukeji.know.streaming.km.collector.sink.kafka;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
+import com.xiaojukeji.know.streaming.km.common.bean.event.metric.ClusterMetricEvent;
+import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.ClusterMetricPO;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
+import org.springframework.context.ApplicationListener;
+import org.springframework.stereotype.Component;
+
+import javax.annotation.PostConstruct;
+
+import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.CLUSTER_INDEX;
+
+@Component
+public class ClusterMetricESSender extends AbstractMetricESSender implements ApplicationListener<ClusterMetricEvent> {
+    private static final ILog  LOGGER = LogFactory.getLog(ClusterMetricESSender.class);
+
+    @PostConstruct
+    public void init(){
+        LOGGER.info("method=init||msg=init finished");
+    }
+
+    @Override
+    public void onApplicationEvent(ClusterMetricEvent event) {
+        send2es(CLUSTER_INDEX, ConvertUtil.list2List(event.getClusterMetrics(), ClusterMetricPO.class));
+    }
+}

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/sink/kafka/GroupMetricESSender.java

@@ -0,0 +1,29 @@
+package com.xiaojukeji.know.streaming.km.collector.sink.kafka;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
+import com.xiaojukeji.know.streaming.km.common.bean.event.metric.GroupMetricEvent;
+import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.GroupMetricPO;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
+import org.springframework.context.ApplicationListener;
+import org.springframework.stereotype.Component;
+
+import javax.annotation.PostConstruct;
+
+import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.GROUP_INDEX;
+
+@Component
+public class GroupMetricESSender extends AbstractMetricESSender implements ApplicationListener<GroupMetricEvent> {
+    private static final ILog  LOGGER = LogFactory.getLog(GroupMetricESSender.class);
+
+    @PostConstruct
+    public void init(){
+        LOGGER.info("method=init||msg=init finished");
+    }
+
+    @Override
+    public void onApplicationEvent(GroupMetricEvent event) {
+        send2es(GROUP_INDEX, ConvertUtil.list2List(event.getGroupMetrics(), GroupMetricPO.class));
+    }
+}

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/sink/kafka/PartitionMetricESSender.java

@@ -0,0 +1,29 @@
+package com.xiaojukeji.know.streaming.km.collector.sink.kafka;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
+import com.xiaojukeji.know.streaming.km.common.bean.event.metric.PartitionMetricEvent;
+import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.PartitionMetricPO;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
+import org.springframework.context.ApplicationListener;
+import org.springframework.stereotype.Component;
+
+import javax.annotation.PostConstruct;
+
+import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.PARTITION_INDEX;
+
+@Component
+public class PartitionMetricESSender extends AbstractMetricESSender implements ApplicationListener<PartitionMetricEvent> {
+    private static final ILog  LOGGER = LogFactory.getLog(PartitionMetricESSender.class);
+
+    @PostConstruct
+    public void init(){
+        LOGGER.info("method=init||msg=init finished");
+    }
+
+    @Override
+    public void onApplicationEvent(PartitionMetricEvent event) {
+        send2es(PARTITION_INDEX, ConvertUtil.list2List(event.getPartitionMetrics(), PartitionMetricPO.class));
+    }
+}

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/sink/kafka/TopicMetricESSender.java

@@ -0,0 +1,29 @@
+package com.xiaojukeji.know.streaming.km.collector.sink.kafka;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
+import com.xiaojukeji.know.streaming.km.common.bean.event.metric.*;
+import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.*;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
+import org.springframework.context.ApplicationListener;
+import org.springframework.stereotype.Component;
+
+import javax.annotation.PostConstruct;
+
+import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.TOPIC_INDEX;
+
+@Component
+public class TopicMetricESSender extends AbstractMetricESSender implements ApplicationListener<TopicMetricEvent> {
+    private static final ILog  LOGGER = LogFactory.getLog(TopicMetricESSender.class);
+
+    @PostConstruct
+    public void init(){
+        LOGGER.info("method=init||msg=init finished");
+    }
+
+    @Override
+    public void onApplicationEvent(TopicMetricEvent event) {
+        send2es(TOPIC_INDEX, ConvertUtil.list2List(event.getTopicMetrics(), TopicMetricPO.class));
+    }
+}

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/sink/kafka/ZookeeperMetricESSender.java

@@ -0,0 +1,29 @@
+package com.xiaojukeji.know.streaming.km.collector.sink.kafka;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
+import com.xiaojukeji.know.streaming.km.common.bean.event.metric.ZookeeperMetricEvent;
+import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.ZookeeperMetricPO;
+import org.springframework.context.ApplicationListener;
+import org.springframework.stereotype.Component;
+
+import javax.annotation.PostConstruct;
+
+import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.ZOOKEEPER_INDEX;
+
+@Component
+public class ZookeeperMetricESSender extends AbstractMetricESSender implements ApplicationListener<ZookeeperMetricEvent> {
+    private static final ILog  LOGGER = LogFactory.getLog(ZookeeperMetricESSender.class);
+
+    @PostConstruct
+    public void init(){
+        LOGGER.info("method=init||msg=init finished");
+    }
+
+    @Override
+    public void onApplicationEvent(ZookeeperMetricEvent event) {
+        send2es(ZOOKEEPER_INDEX, ConvertUtil.list2List(event.getZookeeperMetrics(), ZookeeperMetricPO.class));
+    }
+}

km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector/sink/mm2/MirrorMakerMetricESSender.java

@@ -0,0 +1,33 @@
+package com.xiaojukeji.know.streaming.km.collector.sink.mm2;
+
+import com.didiglobal.logi.log.ILog;
+import com.didiglobal.logi.log.LogFactory;
+import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
+import com.xiaojukeji.know.streaming.km.common.bean.event.metric.mm2.MirrorMakerMetricEvent;
+import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.mm2.MirrorMakerMetricPO;
+import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
+import org.springframework.context.ApplicationListener;
+import org.springframework.stereotype.Component;
+
+import javax.annotation.PostConstruct;
+
+import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.CONNECT_MM2_INDEX;
+
+/**
+ * @author zengqiao
+ * @date 2022/12/20
+ */
+@Component
+public class MirrorMakerMetricESSender extends AbstractMetricESSender implements ApplicationListener<MirrorMakerMetricEvent> {
+    protected static final ILog LOGGER = LogFactory.getLog(MirrorMakerMetricESSender.class);
+
+    @PostConstruct
+    public void init(){
+        LOGGER.info("method=init||msg=init finished");
+    }
+
+    @Override
+    public void onApplicationEvent(MirrorMakerMetricEvent event) {
+        send2es(CONNECT_MM2_INDEX, ConvertUtil.list2List(event.getMetricsList(), MirrorMakerMetricPO.class));
+    }
+}

km-common/pom.xml

@@ -81,10 +81,6 @@
             <version>3.0.2</version>
         </dependency>
 
-        <dependency>
-            <groupId>junit</groupId>
-            <artifactId>junit</artifactId>
-        </dependency>
         <dependency>
             <groupId>org.projectlombok</groupId>
             <artifactId>lombok</artifactId>
@@ -127,5 +123,9 @@
             <groupId>org.apache.kafka</groupId>
             <artifactId>kafka_2.13</artifactId>
         </dependency>
+        <dependency>
+            <groupId>org.apache.kafka</groupId>
+            <artifactId>connect-runtime</artifactId>
+        </dependency>
     </dependencies>
 </project>

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/cluster/ClusterConnectorsOverviewDTO.java

@@ -0,0 +1,28 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.cluster;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.MetricDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationSortDTO;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.Data;
+
+import javax.validation.constraints.NotNull;
+import java.util.List;
+
+
+/**
+ * @author zengqiao
+ * @date 22/02/24
+ */
+@Data
+public class ClusterConnectorsOverviewDTO extends PaginationSortDTO {
+    @NotNull(message = "latestMetricNames不允许为空")
+    @ApiModelProperty("需要指标点的信息")
+    private List<String> latestMetricNames;
+
+    @NotNull(message = "metricLines不允许为空")
+    @ApiModelProperty("需要指标曲线的信息")
+    private MetricDTO metricLines;
+
+    @ApiModelProperty("需要排序的指标名称列表,比较第一个不为空的metric")
+    private List<String> sortMetricNameList;
+}

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/cluster/ClusterGroupSummaryDTO.java

@@ -1,19 +1,18 @@
 package com.xiaojukeji.know.streaming.km.common.bean.dto.cluster;
 
-import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationMulFuzzySearchDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationBaseDTO;
 import io.swagger.annotations.ApiModelProperty;
 import lombok.Data;
 
-
 /**
- * @author zengqiao
+ * @author wyb
- * @date 22/02/24
+ * @date 2022/10/17
  */
 @Data
-public class ClusterGroupsOverviewDTO extends PaginationMulFuzzySearchDTO {
+public class ClusterGroupSummaryDTO extends PaginationBaseDTO {
     @ApiModelProperty("查找该Topic")
-    private String topicName;
+    private String searchTopicName;
 
     @ApiModelProperty("查找该Group")
-    private String groupName;
+    private String searchGroupName;
 }

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/cluster/ClusterMirrorMakersOverviewDTO.java

@@ -0,0 +1,12 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.cluster;
+
+import lombok.Data;
+
+
+/**
+ * @author zengqiao
+ * @date 22/12/12
+ */
+@Data
+public class ClusterMirrorMakersOverviewDTO extends ClusterConnectorsOverviewDTO {
+}

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/cluster/ClusterPhyBaseDTO.java

@@ -3,6 +3,7 @@ package com.xiaojukeji.know.streaming.km.common.bean.dto.cluster;
 import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
 import com.xiaojukeji.know.streaming.km.common.bean.dto.BaseDTO;
 import com.xiaojukeji.know.streaming.km.common.bean.entity.config.JmxConfig;
+import com.xiaojukeji.know.streaming.km.common.bean.entity.config.ZKConfig;
 import io.swagger.annotations.ApiModel;
 import io.swagger.annotations.ApiModelProperty;
 import lombok.Data;
@@ -34,4 +35,8 @@ public class ClusterPhyBaseDTO extends BaseDTO {
     @NotNull(message = "jmxProperties不允许为空")
     @ApiModelProperty(value="Jmx配置")
     protected JmxConfig jmxProperties;
+
+    // TODO 前端页面增加时，需要加一个不为空的限制
+    @ApiModelProperty(value="ZK配置")
+    protected ZKConfig zkProperties;
 }

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/cluster/ClusterZookeepersOverviewDTO.java

@@ -0,0 +1,13 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.cluster;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationBaseDTO;
+import lombok.Data;
+
+/**
+ * @author wyc
+ * @date 2022/9/23
+ */
+@Data
+public class ClusterZookeepersOverviewDTO extends PaginationBaseDTO {
+
+}

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/connect/ClusterConnectorDTO.java

@@ -0,0 +1,32 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.connect;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.BaseDTO;
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+
+import javax.validation.constraints.NotBlank;
+import javax.validation.constraints.NotNull;
+
+/**
+ * @author zengqiao
+ * @date 2022-10-17
+ */
+@Data
+@NoArgsConstructor
+@ApiModel(description = "集群Connector")
+public class ClusterConnectorDTO extends BaseDTO {
+    @NotNull(message = "connectClusterId不允许为空")
+    @ApiModelProperty(value = "Connector集群ID", example = "1")
+    protected Long connectClusterId;
+
+    @NotBlank(message = "name不允许为空串")
+    @ApiModelProperty(value = "Connector名称", example = "know-streaming-connector")
+    protected String connectorName;
+
+    public ClusterConnectorDTO(Long connectClusterId, String connectorName) {
+        this.connectClusterId = connectClusterId;
+        this.connectorName = connectorName;
+    }
+}

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/connect/cluster/ConnectClusterDTO.java

@@ -0,0 +1,29 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.connect.cluster;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.BaseDTO;
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.Data;
+
+/**
+ * @author zengqiao
+ * @date 2022-10-17
+ */
+@Data
+@ApiModel(description = "集群Connector")
+public class ConnectClusterDTO extends BaseDTO {
+    @ApiModelProperty(value = "Connect集群ID", example = "1")
+    private Long id;
+
+    @ApiModelProperty(value = "Connect集群名称", example = "know-streaming")
+    private String name;
+
+    @ApiModelProperty(value = "Connect集群URL", example = "http://127.0.0.1:8080")
+    private String clusterUrl;
+
+    @ApiModelProperty(value = "Connect集群版本", example = "2.5.1")
+    private String version;
+
+    @ApiModelProperty(value = "JMX配置", example = "")
+    private String jmxProperties;
+}

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/connect/connector/ConnectorActionDTO.java

@@ -0,0 +1,20 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.ClusterConnectorDTO;
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.Data;
+
+import javax.validation.constraints.NotBlank;
+
+/**
+ * @author zengqiao
+ * @date 2022-10-17
+ */
+@Data
+@ApiModel(description = "操作Connector")
+public class ConnectorActionDTO extends ClusterConnectorDTO {
+    @NotBlank(message = "action不允许为空串")
+    @ApiModelProperty(value = "Connector名称", example = "stop|restart|resume")
+    private String action;
+}

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/connect/connector/ConnectorCreateDTO.java

@@ -0,0 +1,28 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.ClusterConnectorDTO;
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+
+import javax.validation.constraints.NotNull;
+import java.util.Properties;
+
+/**
+ * @author zengqiao
+ * @date 2022-10-17
+ */
+@Data
+@NoArgsConstructor
+@ApiModel(description = "创建Connector")
+public class ConnectorCreateDTO extends ClusterConnectorDTO {
+    @NotNull(message = "configs不允许为空")
+    @ApiModelProperty(value = "配置", example = "")
+    protected Properties configs;
+
+    public ConnectorCreateDTO(Long connectClusterId, String connectorName, Properties configs) {
+        super(connectClusterId, connectorName);
+        this.configs = configs;
+    }
+}

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/connect/connector/ConnectorDeleteDTO.java

@@ -0,0 +1,14 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.ClusterConnectorDTO;
+import io.swagger.annotations.ApiModel;
+import lombok.Data;
+
+/**
+ * @author zengqiao
+ * @date 2022-10-17
+ */
+@Data
+@ApiModel(description = "删除Connector")
+public class ConnectorDeleteDTO extends ClusterConnectorDTO {
+}

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/connect/mm2/MirrorMaker2ActionDTO.java

@@ -0,0 +1,15 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.connect.mm2;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector.ConnectorActionDTO;
+import io.swagger.annotations.ApiModel;
+import lombok.Data;
+
+
+/**
+ * @author zengqiao
+ * @date 2022-12-12
+ */
+@Data
+@ApiModel(description = "操作MM2")
+public class MirrorMaker2ActionDTO extends ConnectorActionDTO {
+}

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/connect/mm2/MirrorMaker2DeleteDTO.java

@@ -0,0 +1,14 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.connect.mm2;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector.ConnectorDeleteDTO;
+import io.swagger.annotations.ApiModel;
+import lombok.Data;
+
+/**
+ * @author zengqiao
+ * @date 2022-12-12
+ */
+@Data
+@ApiModel(description = "删除MM2")
+public class MirrorMaker2DeleteDTO extends ConnectorDeleteDTO {
+}

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/connect/mm2/MirrorMakerCreateDTO.java

@@ -0,0 +1,69 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.connect.mm2;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector.ConnectorCreateDTO;
+import com.xiaojukeji.know.streaming.km.common.constant.connect.KafkaConnectConstant;
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.Data;
+import org.apache.kafka.clients.CommonClientConfigs;
+
+import javax.validation.Valid;
+import javax.validation.constraints.NotNull;
+import java.util.Properties;
+
+/**
+ * @author zengqiao
+ * @date 2022-12-12
+ */
+@Data
+@ApiModel(description = "创建MM2")
+public class MirrorMakerCreateDTO extends ConnectorCreateDTO {
+    @NotNull(message = "sourceKafkaClusterId不允许为空")
+    @ApiModelProperty(value = "源Kafka集群ID", example = "")
+    private Long sourceKafkaClusterId;
+
+    @Valid
+    @ApiModelProperty(value = "heartbeat-connector的信息", example = "")
+    private Properties heartbeatConnectorConfigs;
+
+    @Valid
+    @ApiModelProperty(value = "checkpoint-connector的信息", example = "")
+    private Properties checkpointConnectorConfigs;
+
+    public void unifyData(Long sourceKafkaClusterId, String sourceBootstrapServers, Properties sourceKafkaProps,
+                          Long targetKafkaClusterId, String targetBootstrapServers, Properties targetKafkaProps) {
+        if (sourceKafkaProps == null) {
+            sourceKafkaProps = new Properties();
+        }
+
+        if (targetKafkaProps == null) {
+            targetKafkaProps = new Properties();
+        }
+
+        this.unifyData(this.configs, sourceKafkaClusterId, sourceBootstrapServers, sourceKafkaProps, targetKafkaClusterId, targetBootstrapServers, targetKafkaProps);
+
+        if (heartbeatConnectorConfigs != null) {
+            this.unifyData(this.heartbeatConnectorConfigs, sourceKafkaClusterId, sourceBootstrapServers, sourceKafkaProps, targetKafkaClusterId, targetBootstrapServers, targetKafkaProps);
+        }
+
+        if (checkpointConnectorConfigs != null) {
+            this.unifyData(this.checkpointConnectorConfigs, sourceKafkaClusterId, sourceBootstrapServers, sourceKafkaProps, targetKafkaClusterId, targetBootstrapServers, targetKafkaProps);
+        }
+    }
+
+    private void unifyData(Properties dataConfig,
+                           Long sourceKafkaClusterId, String sourceBootstrapServers, Properties sourceKafkaProps,
+                           Long targetKafkaClusterId, String targetBootstrapServers, Properties targetKafkaProps) {
+        dataConfig.put(KafkaConnectConstant.MIRROR_MAKER_SOURCE_CLUSTER_ALIAS_FIELD_NAME, sourceKafkaClusterId);
+        dataConfig.put(KafkaConnectConstant.MIRROR_MAKER_SOURCE_CLUSTER_FIELD_NAME + "." + CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG, sourceBootstrapServers);
+        for (Object configKey: sourceKafkaProps.keySet()) {
+            dataConfig.put(KafkaConnectConstant.MIRROR_MAKER_SOURCE_CLUSTER_FIELD_NAME + "." + configKey, sourceKafkaProps.getProperty((String) configKey));
+        }
+
+        dataConfig.put(KafkaConnectConstant.MIRROR_MAKER_TARGET_CLUSTER_ALIAS_FIELD_NAME, targetKafkaClusterId);
+        dataConfig.put(KafkaConnectConstant.MIRROR_MAKER_TARGET_CLUSTER_FIELD_NAME + "." + CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG, targetBootstrapServers);
+        for (Object configKey: targetKafkaProps.keySet()) {
+            dataConfig.put(KafkaConnectConstant.MIRROR_MAKER_TARGET_CLUSTER_FIELD_NAME + "." + configKey, targetKafkaProps.getProperty((String) configKey));
+        }
+    }
+}

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/connect/task/TaskActionDTO.java

@@ -0,0 +1,20 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.connect.task;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector.ConnectorActionDTO;
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.Data;
+
+import javax.validation.constraints.NotNull;
+
+/**
+ * @author zengqiao
+ * @date 2022-10-17
+ */
+@Data
+@ApiModel(description = "操作Task")
+public class TaskActionDTO extends ConnectorActionDTO {
+    @NotNull(message = "taskId不允许为NULL")
+    @ApiModelProperty(value = "taskId", example = "123")
+    private Long taskId;
+}

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/ha/mirror/MirrorTopicCreateDTO.java

@@ -0,0 +1,38 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.ha.mirror;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.BaseDTO;
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.Data;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.validation.constraints.NotNull;
+
+/**
+ * @author zengqiao
+ * @date 20/4/23
+ */
+@Data
+@ApiModel(description="Topic镜像信息")
+public class MirrorTopicCreateDTO extends BaseDTO {
+    @Min(value = 0, message = "sourceClusterPhyId不允许为空，且最小值为0")
+    @ApiModelProperty(value = "源集群ID", example = "3")
+    private Long sourceClusterPhyId;
+
+    @Min(value = 0, message = "destClusterPhyId不允许为空，且最小值为0")
+    @ApiModelProperty(value = "目标集群ID", example = "3")
+    private Long destClusterPhyId;
+
+    @NotBlank(message = "topicName不允许为空串")
+    @ApiModelProperty(value = "Topic名称", example = "mirrorTopic")
+    private String topicName;
+
+    @NotNull(message = "syncData不允许为空")
+    @ApiModelProperty(value = "同步数据", example = "true")
+    private Boolean syncData;
+
+    @NotNull(message = "syncConfig不允许为空")
+    @ApiModelProperty(value = "同步配置", example = "false")
+    private Boolean syncConfig;
+}

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/ha/mirror/MirrorTopicDeleteDTO.java

@@ -0,0 +1,29 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.ha.mirror;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.BaseDTO;
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.Data;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+
+/**
+ * @author zengqiao
+ * @date 20/4/23
+ */
+@Data
+@ApiModel(description="Topic镜像信息")
+public class MirrorTopicDeleteDTO extends BaseDTO {
+    @Min(value = 0, message = "sourceClusterPhyId不允许为空，且最小值为0")
+    @ApiModelProperty(value = "源集群ID", example = "3")
+    private Long sourceClusterPhyId;
+
+    @Min(value = 0, message = "destClusterPhyId不允许为空，且最小值为0")
+    @ApiModelProperty(value = "目标集群ID", example = "3")
+    private Long destClusterPhyId;
+
+    @NotBlank(message = "topicName不允许为空串")
+    @ApiModelProperty(value = "Topic名称", example = "mirrorTopic")
+    private String topicName;
+}

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/metrices/MetricDetailDTO.java

@@ -7,6 +7,9 @@ import lombok.AllArgsConstructor;
 import lombok.Data;
 import lombok.NoArgsConstructor;
 
+import javax.validation.constraints.NotNull;
+
+
 /**
  * @author didi
  */
@@ -22,6 +25,7 @@ public class MetricDetailDTO extends BaseDTO {
     @ApiModelProperty("指标是否显示")
     private Boolean set;
 
+    @NotNull(message = "MetricDetailDTO的rank字段应不为空")
     @ApiModelProperty("指标优先级")
     private Integer rank;
 

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/metrices/UserMetricConfigDTO.java

@@ -7,6 +7,7 @@ import lombok.AllArgsConstructor;
 import lombok.Data;
 import lombok.NoArgsConstructor;
 
+import javax.validation.Valid;
 import java.util.List;
 import java.util.Map;
 
@@ -19,6 +20,7 @@ public class UserMetricConfigDTO extends BaseDTO {
     @ApiModelProperty("指标展示设置项，key：指标名；value：是否展现(true展现/false不展现)")
     private Map<String, Boolean> metricsSet;
 
+    @Valid
     @ApiModelProperty("指标自定义属性列表")
     private List<MetricDetailDTO> metricDetailDTOList;
 }

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/metrices/connect/MetricsConnectClustersDTO.java

@@ -0,0 +1,22 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.connect;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.MetricDTO;
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.AllArgsConstructor;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+
+import java.util.List;
+
+/**
+ * @author didi
+ */
+@Data
+@NoArgsConstructor
+@AllArgsConstructor
+@ApiModel(description = "Connect集群指标查询信息")
+public class MetricsConnectClustersDTO extends MetricDTO {
+    @ApiModelProperty("Connect集群ID")
+    private List<Long> connectClusterIdList;
+}

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/metrices/connect/MetricsConnectorsDTO.java

@@ -0,0 +1,23 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.connect;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.ClusterConnectorDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.MetricDTO;
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.AllArgsConstructor;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+
+import java.util.List;
+
+/**
+ * @author didi
+ */
+@Data
+@NoArgsConstructor
+@AllArgsConstructor
+@ApiModel(description = "Connector指标查询信息")
+public class MetricsConnectorsDTO extends MetricDTO {
+    @ApiModelProperty("Connector列表")
+    private List<ClusterConnectorDTO> connectorNameList;
+}

km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/bean/dto/metrices/mm2/MetricsMirrorMakersDTO.java

@@ -0,0 +1,23 @@
+package com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.mm2;
+
+import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.ClusterConnectorDTO;
+import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.MetricDTO;
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.AllArgsConstructor;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+
+import java.util.List;
+
+/**
+ * @author didi
+ */
+@Data
+@NoArgsConstructor
+@AllArgsConstructor
+@ApiModel(description = "MirrorMaker指标查询信息")
+public class MetricsMirrorMakersDTO extends MetricDTO {
+    @ApiModelProperty("MirrorMaker的SourceConnect列表")
+    private List<ClusterConnectorDTO> connectorNameList;
+}