mirror of
https://github.com/didi/KnowStreaming.git
synced 2025-12-24 20:22:12 +08:00
Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
045f65204b |
51
.github/ISSUE_TEMPLATE/bug_report.md
vendored
51
.github/ISSUE_TEMPLATE/bug_report.md
vendored
@@ -1,51 +0,0 @@
|
||||
---
|
||||
name: 报告Bug
|
||||
about: 报告KnowStreaming的相关Bug
|
||||
title: ''
|
||||
labels: bug
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
- [ ] 我已经在 [issues](https://github.com/didi/KnowStreaming/issues) 搜索过相关问题了,并没有重复的。
|
||||
|
||||
你是否希望来认领这个Bug。
|
||||
|
||||
「 Y / N 」
|
||||
|
||||
### 环境信息
|
||||
|
||||
* KnowStreaming version : <font size=4 color =red> xxx </font>
|
||||
* Operating System version : <font size=4 color =red> xxx </font>
|
||||
* Java version : <font size=4 color =red> xxx </font>
|
||||
|
||||
|
||||
### 重现该问题的步骤
|
||||
|
||||
1. xxx
|
||||
|
||||
|
||||
|
||||
2. xxx
|
||||
|
||||
|
||||
3. xxx
|
||||
|
||||
|
||||
|
||||
### 预期结果
|
||||
|
||||
<!-- 写下应该出现的预期结果？-->
|
||||
|
||||
### 实际结果
|
||||
|
||||
<!-- 实际发生了什么? -->
|
||||
|
||||
|
||||
---
|
||||
|
||||
如果有异常，请附上异常Trace:
|
||||
|
||||
```
|
||||
Just put your stack trace here!
|
||||
```
|
||||
8
.github/ISSUE_TEMPLATE/config.yml
vendored
8
.github/ISSUE_TEMPLATE/config.yml
vendored
@@ -1,8 +0,0 @@
|
||||
blank_issues_enabled: true
|
||||
contact_links:
|
||||
- name: 讨论问题
|
||||
url: https://github.com/didi/KnowStreaming/discussions/new
|
||||
about: 发起问题、讨论 等等
|
||||
- name: KnowStreaming官网
|
||||
url: https://knowstreaming.com/
|
||||
about: KnowStreaming website
|
||||
26
.github/ISSUE_TEMPLATE/detail_optimizing.md
vendored
26
.github/ISSUE_TEMPLATE/detail_optimizing.md
vendored
@@ -1,26 +0,0 @@
|
||||
---
|
||||
name: 优化建议
|
||||
about: 相关功能优化建议
|
||||
title: ''
|
||||
labels: Optimization Suggestions
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
- [ ] 我已经在 [issues](https://github.com/didi/KnowStreaming/issues) 搜索过相关问题了,并没有重复的。
|
||||
|
||||
你是否希望来认领这个优化建议。
|
||||
|
||||
「 Y / N 」
|
||||
|
||||
### 环境信息
|
||||
|
||||
* KnowStreaming version : <font size=4 color =red> xxx </font>
|
||||
* Operating System version : <font size=4 color =red> xxx </font>
|
||||
* Java version : <font size=4 color =red> xxx </font>
|
||||
|
||||
### 需要优化的功能点
|
||||
|
||||
|
||||
### 建议如何优化
|
||||
|
||||
20
.github/ISSUE_TEMPLATE/feature_request.md
vendored
20
.github/ISSUE_TEMPLATE/feature_request.md
vendored
@@ -1,20 +0,0 @@
|
||||
---
|
||||
name: 提议新功能/需求
|
||||
about: 给KnowStreaming提一个功能需求
|
||||
title: ''
|
||||
labels: feature
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
- [ ] 我在 [issues](https://github.com/didi/KnowStreaming/issues) 中并未搜索到与此相关的功能需求。
|
||||
- [ ] 我在 [release note](https://github.com/didi/KnowStreaming/releases) 已经发布的版本中并没有搜到相关功能.
|
||||
|
||||
你是否希望来认领这个Feature。
|
||||
|
||||
「 Y / N 」
|
||||
|
||||
|
||||
## 这里描述需求
|
||||
<!--请尽可能的描述清楚您的需求 -->
|
||||
|
||||
12
.github/ISSUE_TEMPLATE/question.md
vendored
12
.github/ISSUE_TEMPLATE/question.md
vendored
@@ -1,12 +0,0 @@
|
||||
---
|
||||
name: 提个问题
|
||||
about: 问KnowStreaming相关问题
|
||||
title: ''
|
||||
labels: question
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
- [ ] 我已经在 [issues](https://github.com/didi/KnowStreaming/issues) 搜索过相关问题了,并没有重复的。
|
||||
|
||||
## 在这里提出你的问题
|
||||
22
.github/PULL_REQUEST_TEMPLATE.md
vendored
22
.github/PULL_REQUEST_TEMPLATE.md
vendored
@@ -1,22 +0,0 @@
|
||||
请不要在没有先创建Issue的情况下创建Pull Request。
|
||||
|
||||
## 变更的目的是什么
|
||||
|
||||
XXXXX
|
||||
|
||||
## 简短的更新日志
|
||||
|
||||
XX
|
||||
|
||||
## 验证这一变化
|
||||
|
||||
XXXX
|
||||
|
||||
请遵循此清单，以帮助我们快速轻松地整合您的贡献：
|
||||
|
||||
* [ ] 确保有针对更改提交的 Github issue（通常在您开始处理之前）。诸如拼写错误之类的琐碎更改不需要 Github issue。您的Pull Request应该只解决这个问题，而不需要进行其他更改—— 一个 PR 解决一个问题。
|
||||
* [ ] 格式化 Pull Request 标题，如[ISSUE #123] support Confluent Schema Registry。 Pull Request 中的每个提交都应该有一个有意义的主题行和正文。
|
||||
* [ ] 编写足够详细的Pull Request描述，以了解Pull Request的作用、方式和原因。
|
||||
* [ ] 编写必要的单元测试来验证您的逻辑更正。如果提交了新功能或重大更改，请记住在test 模块中添加 integration-test
|
||||
* [ ] 确保编译通过，集成测试通过
|
||||
|
||||
6
.gitignore
vendored
6
.gitignore
vendored
@@ -109,8 +109,4 @@ out/*
|
||||
dist/
|
||||
dist/*
|
||||
km-rest/src/main/resources/templates/
|
||||
*dependency-reduced-pom*
|
||||
#filter flattened xml
|
||||
*/.flattened-pom.xml
|
||||
.flattened-pom.xml
|
||||
*/*/.flattened-pom.xml
|
||||
*dependency-reduced-pom*
|
||||
@@ -1,74 +0,0 @@
|
||||
|
||||
# Contributor Covenant Code of Conduct
|
||||
|
||||
## Our Pledge
|
||||
|
||||
In the interest of fostering an open and welcoming environment, we as
|
||||
contributors and maintainers pledge to making participation in our project and
|
||||
our community a harassment-free experience for everyone, regardless of age, body
|
||||
size, disability, ethnicity, gender identity and expression, level of experience,
|
||||
education, socio-economic status, nationality, personal appearance, race,
|
||||
religion, or sexual identity and orientation.
|
||||
|
||||
## Our Standards
|
||||
|
||||
Examples of behavior that contributes to creating a positive environment
|
||||
include:
|
||||
|
||||
* Using welcoming and inclusive language
|
||||
* Being respectful of differing viewpoints and experiences
|
||||
* Gracefully accepting constructive criticism
|
||||
* Focusing on what is best for the community
|
||||
* Showing empathy towards other community members
|
||||
|
||||
Examples of unacceptable behavior by participants include:
|
||||
|
||||
* The use of sexualized language or imagery and unwelcome sexual attention or
|
||||
advances
|
||||
* Trolling, insulting/derogatory comments, and personal or political attacks
|
||||
* Public or private harassment
|
||||
* Publishing others' private information, such as a physical or electronic
|
||||
address, without explicit permission
|
||||
* Other conduct which could reasonably be considered inappropriate in a
|
||||
professional setting
|
||||
|
||||
## Our Responsibilities
|
||||
|
||||
Project maintainers are responsible for clarifying the standards of acceptable
|
||||
behavior and are expected to take appropriate and fair corrective action in
|
||||
response to any instances of unacceptable behavior.
|
||||
|
||||
Project maintainers have the right and responsibility to remove, edit, or
|
||||
reject comments, commits, code, wiki edits, issues, and other contributions
|
||||
that are not aligned to this Code of Conduct, or to ban temporarily or
|
||||
permanently any contributor for other behaviors that they deem inappropriate,
|
||||
threatening, offensive, or harmful.
|
||||
|
||||
## Scope
|
||||
|
||||
This Code of Conduct applies both within project spaces and in public spaces
|
||||
when an individual is representing the project or its community. Examples of
|
||||
representing a project or community include using an official project e-mail
|
||||
address, posting via an official social media account, or acting as an appointed
|
||||
representative at an online or offline event. Representation of a project may be
|
||||
further defined and clarified by project maintainers.
|
||||
|
||||
## Enforcement
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
||||
reported by contacting the project team at shirenchuang@didiglobal.com . All
|
||||
complaints will be reviewed and investigated and will result in a response that
|
||||
is deemed necessary and appropriate to the circumstances. The project team is
|
||||
obligated to maintain confidentiality with regard to the reporter of an incident.
|
||||
Further details of specific enforcement policies may be posted separately.
|
||||
|
||||
Project maintainers who do not follow or enforce the Code of Conduct in good
|
||||
faith may face temporary or permanent repercussions as determined by other
|
||||
members of the project's leadership.
|
||||
|
||||
## Attribution
|
||||
|
||||
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
|
||||
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
|
||||
|
||||
[homepage]: https://www.contributor-covenant.org
|
||||
158
CONTRIBUTING.md
158
CONTRIBUTING.md
@@ -1,150 +1,28 @@
|
||||
# Contribution Guideline
|
||||
|
||||
Thanks for considering to contribute this project. All issues and pull requests are highly appreciated.
|
||||
|
||||
## Pull Requests
|
||||
|
||||
# 为KnowStreaming做贡献
|
||||
Before sending pull request to this project, please read and follow guidelines below.
|
||||
|
||||
1. Branch: We only accept pull request on `dev` branch.
|
||||
2. Coding style: Follow the coding style used in LogiKM.
|
||||
3. Commit message: Use English and be aware of your spell.
|
||||
4. Test: Make sure to test your code.
|
||||
|
||||
欢迎👏🏻来到KnowStreaming！本文档是关于如何为KnowStreaming做出贡献的指南。
|
||||
Add device mode, API version, related log, screenshots and other related information in your pull request if possible.
|
||||
|
||||
如果您发现不正确或遗漏的内容, 请留下意见/建议。
|
||||
NOTE: We assume all your contribution can be licensed under the [AGPL-3.0](LICENSE).
|
||||
|
||||
## 行为守则
|
||||
请务必阅读并遵守我们的 [行为准则](./CODE_OF_CONDUCT.md).
|
||||
## Issues
|
||||
|
||||
We love clearly described issues. :)
|
||||
|
||||
Following information can help us to resolve the issue faster.
|
||||
|
||||
## 贡献
|
||||
|
||||
**KnowStreaming** 欢迎任何角色的新参与者，包括 **User** 、**Contributor**、**Committer**、**PMC** 。
|
||||
|
||||
我们鼓励新人积极加入 **KnowStreaming** 项目，从User到Contributor、Committer ，甚至是 PMC 角色。
|
||||
|
||||
为了做到这一点，新人需要积极地为 **KnowStreaming** 项目做出贡献。以下介绍如何对 **KnowStreaming** 进行贡献。
|
||||
|
||||
|
||||
### 创建/打开 Issue
|
||||
|
||||
如果您在文档中发现拼写错误、在代码中**发现错误**或想要**新功能**或想要**提供建议**，您可以在 GitHub 上[创建一个Issue](https://github.com/didi/KnowStreaming/issues/new/choose) 进行报告。
|
||||
|
||||
|
||||
如果您想直接贡献, 您可以选择下面标签的问题。
|
||||
|
||||
- [contribution welcome](https://github.com/didi/KnowStreaming/labels/contribution%20welcome) : 非常需要解决/新增 的Issues
|
||||
- [good first issue](https://github.com/didi/KnowStreaming/labels/good%20first%20issue): 对新人比较友好, 新人可以拿这个Issue来练练手热热身。
|
||||
|
||||
<font color=red ><b> 请注意，任何 PR 都必须与有效issue相关联。否则，PR 将被拒绝。</b></font>
|
||||
|
||||
|
||||
|
||||
### 开始你的贡献
|
||||
|
||||
**分支介绍**
|
||||
|
||||
我们将 `dev`分支作为开发分支, 说明这是一个不稳定的分支。
|
||||
|
||||
此外,我们的分支模型符合 [https://nvie.com/posts/a-successful-git-branching-model/](https://nvie.com/posts/a-successful-git-branching-model/). 我们强烈建议新人在创建PR之前先阅读上述文章。
|
||||
|
||||
|
||||
|
||||
**贡献流程**
|
||||
|
||||
为方便描述,我们这里定义一下2个名词：
|
||||
|
||||
自己Fork出来的仓库是私人仓库, 我们这里称之为 ：**分叉仓库**
|
||||
Fork的源项目,我们称之为：**源仓库**
|
||||
|
||||
|
||||
现在，如果您准备好创建PR, 以下是贡献者的工作流程:
|
||||
|
||||
1. Fork [KnowStreaming](https://github.com/didi/KnowStreaming) 项目到自己的仓库
|
||||
|
||||
2. 从源仓库的`dev`拉取并创建自己的本地分支,例如: `dev`
|
||||
3. 在本地分支上对代码进行修改
|
||||
4. Rebase 开发分支, 并解决冲突
|
||||
5. commit 并 push 您的更改到您自己的**分叉仓库**
|
||||
6. 创建一个 Pull Request 到**源仓库**的`dev`分支中。
|
||||
7. 等待回复。如果回复的慢，请无情的催促。
|
||||
|
||||
|
||||
更为详细的贡献流程请看：[贡献流程](./docs/contributer_guide/贡献流程.md)
|
||||
|
||||
创建Pull Request时：
|
||||
|
||||
1. 请遵循 PR的 [模板](./.github/PULL_REQUEST_TEMPLATE.md)
|
||||
2. 请确保 PR 有相应的issue。
|
||||
3. 如果您的 PR 包含较大的更改，例如组件重构或新组件，请编写有关其设计和使用的详细文档(在对应的issue中)。
|
||||
4. 注意单个 PR 不能太大。如果需要进行大量更改，最好将更改分成几个单独的 PR。
|
||||
5. 在合并PR之前，尽量的将最终的提交信息清晰简洁, 将多次修改的提交尽可能的合并为一次提交。
|
||||
6. 创建 PR 后，将为PR分配一个或多个reviewers。
|
||||
|
||||
|
||||
<font color=red><b>如果您的 PR 包含较大的更改，例如组件重构或新组件，请编写有关其设计和使用的详细文档。</b></font>
|
||||
|
||||
|
||||
# 代码审查指南
|
||||
|
||||
Commiter将轮流review代码，以确保在合并前至少有一名Commiter
|
||||
|
||||
一些原则：
|
||||
|
||||
- 可读性——重要的代码应该有详细的文档。API 应该有 Javadoc。代码风格应与现有风格保持一致。
|
||||
- 优雅：新的函数、类或组件应该设计得很好。
|
||||
- 可测试性——单元测试用例应该覆盖 80% 的新代码。
|
||||
- 可维护性 - 遵守我们的编码规范。
|
||||
|
||||
|
||||
# 开发者
|
||||
|
||||
## 成为Contributor
|
||||
|
||||
只要成功提交并合并PR , 则为Contributor
|
||||
|
||||
贡献者名单请看：[贡献者名单](./docs/contributer_guide/开发者名单.md)
|
||||
|
||||
## 尝试成为Commiter
|
||||
|
||||
一般来说, 贡献8个重要的补丁并至少让三个不同的人来Review他们(您需要3个Commiter的支持)。
|
||||
然后请人给你提名, 您需要展示您的
|
||||
|
||||
1. 至少8个重要的PR和项目的相关问题
|
||||
2. 与团队合作的能力
|
||||
3. 了解项目的代码库和编码风格
|
||||
4. 编写好代码的能力
|
||||
|
||||
当前的Commiter可以通过在KnowStreaming中的Issue标签 `nomination`(提名)来提名您
|
||||
|
||||
1. 你的名字和姓氏
|
||||
2. 指向您的Git个人资料的链接
|
||||
3. 解释为什么你应该成为Commiter
|
||||
4. 详细说明提名人与您合作的3个PR以及相关问题,这些问题可以证明您的能力。
|
||||
|
||||
另外2个Commiter需要支持您的**提名**，如果5个工作日内没有人反对，您就是提交者,如果有人反对或者想要更多的信息，Commiter会讨论并通常达成共识(5个工作日内) 。
|
||||
|
||||
|
||||
# 开源奖励计划
|
||||
|
||||
|
||||
我们非常欢迎开发者们为KnowStreaming开源项目贡献一份力量，相应也将给予贡献者激励以表认可与感谢。
|
||||
|
||||
|
||||
## 参与贡献
|
||||
|
||||
1. 积极参与 Issue 的讨论，如答疑解惑、提供想法或报告无法解决的错误（Issue）
|
||||
2. 撰写和改进项目的文档（Wiki）
|
||||
3. 提交补丁优化代码（Coding）
|
||||
|
||||
|
||||
## 你将获得
|
||||
|
||||
1. 加入KnowStreaming开源项目贡献者名单并展示
|
||||
2. KnowStreaming开源贡献者证书(纸质&电子版)
|
||||
3. KnowStreaming贡献者精美大礼包(KnowStreamin/滴滴 周边)
|
||||
|
||||
|
||||
## 相关规则
|
||||
|
||||
- Contributer和Commiter都会有对应的证书和对应的礼包
|
||||
- 每季度有KnowStreaming项目团队评选出杰出贡献者,颁发相应证书。
|
||||
- 年末进行年度评选
|
||||
|
||||
贡献者名单请看：[贡献者名单](./docs/contributer_guide/开发者名单.md)
|
||||
* Device mode and hardware information.
|
||||
* API version.
|
||||
* Logs.
|
||||
* Screenshots.
|
||||
* Steps to reproduce the issue.
|
||||
22
README.md
22
README.md
@@ -45,14 +45,7 @@
|
||||
|
||||
## `Know Streaming` 简介
|
||||
|
||||
`Know Streaming`是一套云原生的Kafka管控平台，脱胎于众多互联网内部多年的Kafka运营实践经验，专注于Kafka运维管控、监控告警、资源治理、多活容灾等核心场景。在用户体验、监控、运维管控上进行了平台化、可视化、智能化的建设，提供一系列特色的功能，极大地方便了用户和运维人员的日常使用，让普通运维人员都能成为Kafka专家。
|
||||
|
||||
我们现在正在收集 Know Streaming 用户信息，以帮助我们进一步改进 Know Streaming。
|
||||
请在 [issue#663](https://github.com/didi/KnowStreaming/issues/663) 上提供您的使用信息来支持我们：[谁在使用 Know Streaming](https://github.com/didi/KnowStreaming/issues/663)
|
||||
|
||||
|
||||
|
||||
整体具有以下特点：
|
||||
`Know Streaming`是一套云原生的Kafka管控平台，脱胎于众多互联网内部多年的Kafka运营实践经验，专注于Kafka运维管控、监控告警、资源治理、多活容灾等核心场景。在用户体验、监控、运维管控上进行了平台化、可视化、智能化的建设，提供一系列特色的功能，极大地方便了用户和运维人员的日常使用，让普通运维人员都能成为Kafka专家。整体具有以下特点：
|
||||
|
||||
- 👀 **零侵入、全覆盖**
|
||||
- 无需侵入改造 `Apache Kafka` ，一键便能纳管 `0.10.x` ~ `3.x.x` 众多版本的Kafka，包括 `ZK` 或 `Raft` 运行模式的版本，同时在兼容架构上具备良好的扩展性，帮助您提升集群管理水平；
|
||||
@@ -106,13 +99,9 @@
|
||||
|
||||
## 成为社区贡献者
|
||||
|
||||
1. [贡献源码](https://doc.knowstreaming.com/product/10-contribution) 了解如何成为 Know Streaming 的贡献者
|
||||
2. [具体贡献流程](https://doc.knowstreaming.com/product/10-contribution#102-贡献流程)
|
||||
3. [开源激励计划](https://doc.knowstreaming.com/product/10-contribution#105-开源激励计划)
|
||||
4. [贡献者名单](https://doc.knowstreaming.com/product/10-contribution#106-贡献者名单)
|
||||
点击 [这里](CONTRIBUTING.md)，了解如何成为 Know Streaming 的贡献者
|
||||
|
||||
|
||||
获取KnowStreaming开源社区证书。
|
||||
|
||||
## 加入技术交流群
|
||||
|
||||
@@ -143,13 +132,8 @@ PS: 提问请尽量把问题一次性描述清楚，并告知环境信息情况
|
||||
|
||||
**`2、微信群`**
|
||||
|
||||
微信加群：添加`mike_zhangliang`、`PenceXie` 、`szzdzhp001`的微信号备注KnowStreaming加群。
|
||||
微信加群：添加`mike_zhangliang`、`PenceXie`的微信号备注KnowStreaming加群。
|
||||
<br/>
|
||||
|
||||
加群之前有劳点一下 star，一个小小的 star 是对KnowStreaming作者们努力建设社区的动力。
|
||||
|
||||
感谢感谢！！！
|
||||
|
||||
<img width="116" alt="wx" src="https://user-images.githubusercontent.com/71620349/192257217-c4ebc16c-3ad9-485d-a914-5911d3a4f46b.png">
|
||||
|
||||
## Star History
|
||||
|
||||
@@ -1,129 +1,4 @@
|
||||
|
||||
## v3.2.0
|
||||
|
||||
**问题修复**
|
||||
- 修复健康巡检结果更新至 DB 时，出现死锁问题；
|
||||
- 修复 KafkaJMXClient 类中，logger错误的问题；
|
||||
- 后端修复 Topic 过期策略在 0.10.1.0 版本能多选的问题，实际应该只能二选一；
|
||||
- 修复接入集群时，不填写集群配置会报错的问题；
|
||||
- 升级 spring-context 至 5.3.19 版本，修复安全漏洞；
|
||||
- 修复 Broker & Topic 修改配置时，多版本兼容配置的版本信息错误的问题；
|
||||
- 修复 Topic 列表的健康分为健康状态；
|
||||
- 修复 Broker LogSize 指标存储名称错误导致查询不到的问题；
|
||||
- 修复 Prometheus 中，缺少 Group 部分指标的问题；
|
||||
- 修复因缺少健康状态指标导致集群数错误的问题；
|
||||
- 修复后台任务记录操作日志时，因缺少操作用户信息导致出现异常的问题；
|
||||
- 修复 Replica 指标查询时，DSL 错误的问题；
|
||||
- 关闭 errorLogger，修复错误日志重复输出的问题；
|
||||
- 修复系统管理更新用户信息失败的问题；
|
||||
- 修复因原AR信息丢失，导致迁移任务一直处于执行中的错误；
|
||||
- 修复集群 Topic 列表实时数据查询时，出现失败的问题；
|
||||
- 修复集群 Topic 列表，页面白屏问题；
|
||||
- 修复副本变更时，因AR数据异常，导致数组访问越界的问题；
|
||||
|
||||
|
||||
**产品优化**
|
||||
- 优化健康巡检为按照资源维度多线程并发处理；
|
||||
- 统一日志输出格式，并优化部分输出的日志；
|
||||
- 优化 ZK 四字命令结果解析过程中，容易引起误解的 WARN 日志；
|
||||
- 优化 Zookeeper 详情中，目录结构的搜索文案；
|
||||
- 优化线程池的名称，方便第三方系统进行相关问题的分析；
|
||||
- 去除 ESClient 的并发访问控制，降低 ESClient 创建数及提升利用率；
|
||||
- 优化 Topic Messages 抽屉文案；
|
||||
- 优化 ZK 健康巡检失败时的错误日志信息；
|
||||
- 提高 Offset 信息获取的超时时间，降低并发过高时出现请求超时的概率；
|
||||
- 优化 Topic & Partition 元信息的更新策略，降低对 DB 连接的占用；
|
||||
- 优化 Sonar 代码扫码问题；
|
||||
- 优化分区 Offset 指标的采集；
|
||||
- 优化前端图表相关组件逻辑;
|
||||
- 优化产品主题色;
|
||||
- Consumer 列表刷新按钮新增 hover 提示；
|
||||
- 优化配置 Topic 的消息大小时的测试弹框体验；
|
||||
- 优化 Overview 页面 TopN 查询的流程；
|
||||
|
||||
|
||||
**功能新增**
|
||||
- 新增页面无数据排查文档；
|
||||
- 增加 ES 索引删除的功能；
|
||||
- 支持拆分API服务和Job服务部署；
|
||||
|
||||
|
||||
**Kafka Connect Beta版 (v3.2.0版本新增发布)**
|
||||
- Connect 集群的纳管；
|
||||
- Connector 的增删改查；
|
||||
- Connect 集群 & Connector 的指标大盘；
|
||||
|
||||
|
||||
---
|
||||
|
||||
|
||||
## v3.1.0
|
||||
|
||||
**Bug修复**
|
||||
- 修复重置 Group Offset 的提示信息中，缺少Dead状态也可进行重置的描述；
|
||||
- 修复新建 Topic 后，立即查看 Topic Messages 信息时，会提示 Topic 不存在的问题；
|
||||
- 修复副本变更时，优先副本选举未被正常处罚执行的问题；
|
||||
- 修复 git 目录不存在时，打包不能正常进行的问题；
|
||||
- 修复 KRaft 模式的 Kafka 集群，JMX PORT 显示 -1 的问题；
|
||||
|
||||
|
||||
**体验优化**
|
||||
- 优化Cluster、Broker、Topic、Group的健康分为健康状态；
|
||||
- 去除健康巡检配置中的权重信息；
|
||||
- 错误提示页面展示优化；
|
||||
- 前端打包编译依赖默认使用 taobao 镜像；
|
||||
- 重新设计优化导航栏的 icon ；
|
||||
|
||||
|
||||
**新增**
|
||||
- 个人头像下拉信息中，新增产品版本信息；
|
||||
- 多集群列表页面，新增集群健康状态分布信息；
|
||||
|
||||
|
||||
**Kafka ZK 部分 (v3.1.0版本正式发布)**
|
||||
- 新增 ZK 集群的指标大盘信息；
|
||||
- 新增 ZK 集群的服务状态概览信息；
|
||||
- 新增 ZK 集群的服务节点列表信息；
|
||||
- 新增 Kafka 在 ZK 的存储数据查看功能；
|
||||
- 新增 ZK 的健康巡检及健康状态计算；
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
|
||||
## v3.0.1
|
||||
|
||||
**Bug修复**
|
||||
- 修复重置 Group Offset 时，提示信息中缺少 Dead 状态也可进行重置的信息；
|
||||
- 修复 Ldap 某个属性不存在时，会直接抛出空指针导致登陆失败的问题；
|
||||
- 修复集群 Topic 列表页，健康分详情信息中，检查时间展示错误的问题；
|
||||
- 修复更新健康检查结果时，出现死锁的问题；
|
||||
- 修复 Replica 索引模版错误的问题；
|
||||
- 修复 FAQ 文档中的错误链接；
|
||||
- 修复 Broker 的 TopN 指标不存在时，页面数据不展示的问题；
|
||||
- 修复 Group 详情页，图表时间范围选择不生效的问题；
|
||||
|
||||
|
||||
**体验优化**
|
||||
- 集群 Group 列表按照 Group 维度进行展示；
|
||||
- 优化避免因 ES 中该指标不存在，导致日志中出现大量空指针的问题；
|
||||
- 优化全局 Message & Notification 展示效果；
|
||||
- 优化 Topic 扩分区名称 & 描述展示；
|
||||
|
||||
|
||||
**新增**
|
||||
- Broker 列表页面，新增 JMX 是否成功连接的信息；
|
||||
|
||||
|
||||
**ZK 部分(未完全发布)**
|
||||
- 后端补充 Kafka ZK 指标采集，Kafka ZK 信息获取相关功能；
|
||||
- 增加本地缓存，避免同一采集周期内 ZK 指标重复采集；
|
||||
- 增加 ZK 节点采集失败跳过策略，避免不断对存在问题的节点不断尝试；
|
||||
- 修复 zkAvgLatency 指标转 Long 时抛出异常问题；
|
||||
- 修复 ks_km_zookeeper 表中，role 字段类型错误问题；
|
||||
|
||||
---
|
||||
|
||||
## v3.0.0
|
||||
|
||||
@@ -150,7 +25,7 @@
|
||||
- 集群信息中，新增 Kafka 集群运行模式字段
|
||||
- 新增 docker-compose 的部署方式
|
||||
|
||||
---
|
||||
|
||||
|
||||
## v3.0.0-beta.3
|
||||
|
||||
|
||||
@@ -439,7 +439,7 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
|
||||
curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: application/json' http://${esaddr}:${port}/_template/ks_kafka_replication_metric -d '{
|
||||
"order" : 10,
|
||||
"index_patterns" : [
|
||||
"ks_kafka_replication_metric*"
|
||||
"ks_kafka_partition_metric*"
|
||||
],
|
||||
"settings" : {
|
||||
"index" : {
|
||||
@@ -500,7 +500,30 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
|
||||
}
|
||||
},
|
||||
"aliases" : { }
|
||||
}'
|
||||
}[root@10-255-0-23 template]# cat ks_kafka_replication_metric
|
||||
PUT _template/ks_kafka_replication_metric
|
||||
{
|
||||
"order" : 10,
|
||||
"index_patterns" : [
|
||||
"ks_kafka_replication_metric*"
|
||||
],
|
||||
"settings" : {
|
||||
"index" : {
|
||||
"number_of_shards" : "10"
|
||||
}
|
||||
},
|
||||
"mappings" : {
|
||||
"properties" : {
|
||||
"timestamp" : {
|
||||
"format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
|
||||
"index" : true,
|
||||
"type" : "date",
|
||||
"doc_values" : true
|
||||
}
|
||||
}
|
||||
},
|
||||
"aliases" : { }
|
||||
}'
|
||||
|
||||
curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: application/json' http://${esaddr}:${port}/_template/ks_kafka_topic_metric -d '{
|
||||
"order" : 10,
|
||||
@@ -617,92 +640,7 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
|
||||
}
|
||||
},
|
||||
"aliases" : { }
|
||||
}'
|
||||
|
||||
curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: application/json' http://${SERVER_ES_ADDRESS}/_template/ks_kafka_zookeeper_metric -d '{
|
||||
"order" : 10,
|
||||
"index_patterns" : [
|
||||
"ks_kafka_zookeeper_metric*"
|
||||
],
|
||||
"settings" : {
|
||||
"index" : {
|
||||
"number_of_shards" : "10"
|
||||
}
|
||||
},
|
||||
"mappings" : {
|
||||
"properties" : {
|
||||
"routingValue" : {
|
||||
"type" : "text",
|
||||
"fields" : {
|
||||
"keyword" : {
|
||||
"ignore_above" : 256,
|
||||
"type" : "keyword"
|
||||
}
|
||||
}
|
||||
},
|
||||
"clusterPhyId" : {
|
||||
"type" : "long"
|
||||
},
|
||||
"metrics" : {
|
||||
"properties" : {
|
||||
"AvgRequestLatency" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"MinRequestLatency" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"MaxRequestLatency" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"OutstandingRequests" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"NodeCount" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"WatchCount" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"NumAliveConnections" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"PacketsReceived" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"PacketsSent" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"EphemeralsCount" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"ApproximateDataSize" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"OpenFileDescriptorCount" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"MaxFileDescriptorCount" : {
|
||||
"type" : "double"
|
||||
}
|
||||
}
|
||||
},
|
||||
"key" : {
|
||||
"type" : "text",
|
||||
"fields" : {
|
||||
"keyword" : {
|
||||
"ignore_above" : 256,
|
||||
"type" : "keyword"
|
||||
}
|
||||
}
|
||||
},
|
||||
"timestamp" : {
|
||||
"format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
|
||||
"type" : "date"
|
||||
}
|
||||
}
|
||||
},
|
||||
"aliases" : { }
|
||||
}'
|
||||
}'
|
||||
|
||||
for i in {0..6};
|
||||
do
|
||||
@@ -712,7 +650,6 @@ do
|
||||
curl -s -o /dev/null -X PUT http://${esaddr}:${port}/ks_kafka_group_metric${logdate} && \
|
||||
curl -s -o /dev/null -X PUT http://${esaddr}:${port}/ks_kafka_partition_metric${logdate} && \
|
||||
curl -s -o /dev/null -X PUT http://${esaddr}:${port}/ks_kafka_replication_metric${logdate} && \
|
||||
curl -s -o /dev/null -X PUT http://${esaddr}:${port}/ks_kafka_zookeeper_metric${logdate} && \
|
||||
curl -s -o /dev/null -X PUT http://${esaddr}:${port}/ks_kafka_topic_metric${logdate} || \
|
||||
exit 2
|
||||
done
|
||||
done
|
||||
@@ -1 +0,0 @@
|
||||
TODO.
|
||||
@@ -1,6 +0,0 @@
|
||||
|
||||
开源贡献者证书发放名单(定期更新)
|
||||
|
||||
|
||||
贡献者名单请看：[贡献者名单](https://doc.knowstreaming.com/product/10-contribution#106-贡献者名单)
|
||||
|
||||
@@ -1,6 +0,0 @@
|
||||
|
||||
|
||||
<br>
|
||||
<br>
|
||||
|
||||
请点击：[贡献流程](https://doc.knowstreaming.com/product/10-contribution#102-贡献流程)
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 63 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 306 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 306 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 17 KiB |
@@ -1,69 +0,0 @@
|
||||
|
||||
## 支持Kerberos认证的ZK
|
||||
|
||||
|
||||
### 1、修改 KnowStreaming 代码
|
||||
|
||||
代码位置：`src/main/java/com/xiaojukeji/know/streaming/km/persistence/kafka/KafkaAdminZKClient.java`
|
||||
|
||||
将 `createZKClient` 的 `135行 的 false 改为 true
|
||||
|
||||
|
||||
|
||||
修改完后重新进行打包编译，打包编译见：[打包编译](https://github.com/didi/KnowStreaming/blob/master/docs/install_guide/%E6%BA%90%E7%A0%81%E7%BC%96%E8%AF%91%E6%89%93%E5%8C%85%E6%89%8B%E5%86%8C.md
|
||||
)
|
||||
|
||||
|
||||
|
||||
### 2、查看用户在ZK的ACL
|
||||
|
||||
假设我们使用的用户是 `kafka` 这个用户。
|
||||
|
||||
- 1、查看 server.properties 的配置的 zookeeper.connect 的地址；
|
||||
- 2、使用 `zkCli.sh -serve zookeeper.connect的地址` 登录到ZK页面；
|
||||
- 3、ZK页面上，执行命令 `getAcl /kafka` 查看 `kafka` 用户的权限；
|
||||
|
||||
此时，我们可以看到如下信息：
|
||||
|
||||
|
||||
`kafka` 用户需要的权限是 `cdrwa`。如果用户没有 `cdrwa` 权限的话，需要创建用户并授权，授权命令为：`setAcl`
|
||||
|
||||
|
||||
### 3、创建Kerberos的keytab并修改 KnowStreaming 主机
|
||||
|
||||
- 1、在 Kerberos 的域中创建 `kafka/_HOST` 的 `keytab`，并导出。例如：`kafka/dbs-kafka-test-8-53`；
|
||||
- 2、导出 keytab 后上传到安装 KS 的机器的 `/etc/keytab` 下；
|
||||
- 3、在 KS 机器上，执行 `kinit -kt zookeepe.keytab kafka/dbs-kafka-test-8-53` 看是否能进行 `Kerberos` 登录；
|
||||
- 4、可以登录后，配置 `/opt/zookeeper.jaas` 文件，例子如下：
|
||||
```sql
|
||||
Client {
|
||||
com.sun.security.auth.module.Krb5LoginModule required
|
||||
useKeyTab=true
|
||||
storeKey=false
|
||||
serviceName="zookeeper"
|
||||
keyTab="/etc/keytab/zookeeper.keytab"
|
||||
principal="kafka/dbs-kafka-test-8-53@XXX.XXX.XXX";
|
||||
};
|
||||
```
|
||||
- 5、需要配置 `KDC-Server` 对 `KnowStreaming` 的机器开通防火墙，并在KS的机器 `/etc/host/` 配置 `kdc-server` 的 `hostname`。并将 `krb5.conf` 导入到 `/etc` 下；
|
||||
|
||||
|
||||
### 4、修改 KnowStreaming 的配置
|
||||
|
||||
- 1、在 `/usr/local/KnowStreaming/KnowStreaming/bin/startup.sh` 中的47行的JAVA_OPT中追加如下设置
|
||||
```bash
|
||||
-Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/opt/zookeeper.jaas
|
||||
```
|
||||
|
||||
- 2、重启KS集群后再 start.out 中看到如下信息，则证明Kerberos配置成功；
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
### 5、补充说明
|
||||
|
||||
- 1、多Kafka集群如果用的是一样的Kerberos域的话，只需在每个`ZK`中给`kafka`用户配置`crdwa`权限即可，这样集群初始化的时候`zkclient`是都可以认证；
|
||||
- 2、当前需要修改代码重新打包才可以支持，后续考虑通过页面支持Kerberos认证的ZK接入；
|
||||
- 3、多个Kerberos域暂时未适配；
|
||||
@@ -1,286 +0,0 @@
|
||||
## 1、集群接入错误
|
||||
|
||||
### 1.1、异常现象
|
||||
|
||||
如下图所示，集群非空时，大概率为地址配置错误导致。
|
||||
|
||||
<img src=http://img-ys011.didistatic.com/static/dc2img/do1_BRiXBvqYFK2dxSF1aqgZ width="80%">
|
||||
|
||||
|
||||
|
||||
### 1.2、解决方案
|
||||
|
||||
接入集群时，依据提示的错误，进行相应的解决。例如：
|
||||
|
||||
<img src=http://img-ys011.didistatic.com/static/dc2img/do1_Yn4LhV8aeSEKX1zrrkUi width="50%">
|
||||
|
||||
### 1.3、正常情况
|
||||
|
||||
接入集群时，页面信息都自动正常出现，没有提示错误。
|
||||
|
||||
|
||||
|
||||
## 2、JMX连接失败（需使用3.0.1及以上版本）
|
||||
|
||||
### 2.1异常现象
|
||||
|
||||
Broker列表的JMX Port列出现红色感叹号，则该Broker的JMX连接异常。
|
||||
|
||||
<img src=http://img-ys011.didistatic.com/static/dc2img/do1_MLlLCfAktne4X6MBtBUd width="90%">
|
||||
|
||||
|
||||
|
||||
#### 2.1.1、原因一：JMX未开启
|
||||
|
||||
##### 2.1.1.1、异常现象
|
||||
|
||||
broker列表的JMX Port值为-1，对应Broker的JMX未开启。
|
||||
|
||||
<img src=http://img-ys011.didistatic.com/static/dc2img/do1_E1PD8tPsMeR2zYLFBFAu width="90%">
|
||||
|
||||
##### 2.1.1.2、解决方案
|
||||
|
||||
开启JMX，开启流程如下：
|
||||
|
||||
1、修改kafka的bin目录下面的：`kafka-server-start.sh`文件
|
||||
|
||||
```
|
||||
# 在这个下面增加JMX端口的配置
|
||||
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
|
||||
export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
|
||||
export JMX_PORT=9999 # 增加这个配置, 这里的数值并不一定是要9999
|
||||
fi
|
||||
```
|
||||
|
||||
|
||||
|
||||
2、修改kafka的bin目录下面对的：`kafka-run-class.sh`文件
|
||||
|
||||
```
|
||||
# JMX settings
|
||||
if [ -z "$KAFKA_JMX_OPTS" ]; then
|
||||
KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false
|
||||
-Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=${当前机器的IP}"
|
||||
fi
|
||||
|
||||
# JMX port to use
|
||||
if [ $JMX_PORT ]; then
|
||||
KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT - Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
|
||||
fi
|
||||
```
|
||||
|
||||
|
||||
|
||||
3、重启Kafka-Broker。
|
||||
|
||||
|
||||
|
||||
#### 2.1.2、原因二：JMX配置错误
|
||||
|
||||
##### 2.1.2.1、异常现象
|
||||
|
||||
错误日志：
|
||||
|
||||
```
|
||||
# 错误一： 错误提示的是真实的IP，这样的话基本就是JMX配置的有问题了。
|
||||
2021-01-27 10:06:20.730 ERROR 50901 --- [ics-Thread-1-62] c.x.k.m.c.utils.jmx.JmxConnectorWrap : JMX connect exception, host:192.168.0.1 port:9999. java.rmi.ConnectException: Connection refused to host: 192.168.0.1; nested exception is:
|
||||
|
||||
# 错误二：错误提示的是127.0.0.1这个IP，这个是机器的hostname配置的可能有问题。
|
||||
2021-01-27 10:06:20.730 ERROR 50901 --- [ics-Thread-1-62] c.x.k.m.c.utils.jmx.JmxConnectorWrap : JMX connect exception, host:127.0.0.1 port:9999. java.rmi.ConnectException: Connection refused to host: 127.0.0.1;; nested exception is:
|
||||
```
|
||||
|
||||
|
||||
|
||||
##### 2.1.2.2、解决方案
|
||||
|
||||
开启JMX，开启流程如下：
|
||||
|
||||
1、修改kafka的bin目录下面的：`kafka-server-start.sh`文件
|
||||
|
||||
```
|
||||
# 在这个下面增加JMX端口的配置
|
||||
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
|
||||
export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
|
||||
export JMX_PORT=9999 # 增加这个配置, 这里的数值并不一定是要9999
|
||||
fi
|
||||
```
|
||||
|
||||
|
||||
|
||||
2、修改kafka的bin目录下面对的：`kafka-run-class.sh`文件
|
||||
|
||||
```
|
||||
# JMX settings
|
||||
if [ -z "$KAFKA_JMX_OPTS" ]; then
|
||||
KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false
|
||||
-Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=${当前机器的IP}"
|
||||
fi
|
||||
|
||||
# JMX port to use
|
||||
if [ $JMX_PORT ]; then
|
||||
KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT - Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
|
||||
fi
|
||||
```
|
||||
|
||||
|
||||
|
||||
3、重启Kafka-Broker。
|
||||
|
||||
|
||||
|
||||
#### 2.1.3、原因三：JMX开启SSL
|
||||
|
||||
##### 2.1.3.1、解决方案
|
||||
|
||||
<img src=http://img-ys011.didistatic.com/static/dc2img/do1_kNyCi8H9wtHSRkWurB6S width="50%">
|
||||
|
||||
#### 2.1.4、原因四：连接了错误IP
|
||||
|
||||
##### 2.1.4.1、异常现象
|
||||
|
||||
Broker 配置了内外网，而JMX在配置时，可能配置了内网IP或者外网IP，此时`KnowStreaming` 需要连接到特定网络的IP才可以进行访问。
|
||||
|
||||
比如：Broker在ZK的存储结构如下所示，我们期望连接到 `endpoints` 中标记为 `INTERNAL` 的地址，但是 `KnowStreaming` 却连接了 `EXTERNAL` 的地址。
|
||||
|
||||
```json
|
||||
{
|
||||
"listener_security_protocol_map": {
|
||||
"EXTERNAL": "SASL_PLAINTEXT",
|
||||
"INTERNAL": "SASL_PLAINTEXT"
|
||||
},
|
||||
"endpoints": [
|
||||
"EXTERNAL://192.168.0.1:7092",
|
||||
"INTERNAL://192.168.0.2:7093"
|
||||
],
|
||||
"jmx_port": 8099,
|
||||
"host": "192.168.0.1",
|
||||
"timestamp": "1627289710439",
|
||||
"port": -1,
|
||||
"version": 4
|
||||
}
|
||||
```
|
||||
|
||||
##### 2.1.4.2、解决方案
|
||||
|
||||
可以手动往`ks_km_physical_cluster`表的`jmx_properties`字段增加一个`useWhichEndpoint`字段，从而控制 `KnowStreaming` 连接到特定的JMX IP及PORT。
|
||||
|
||||
`jmx_properties`格式：
|
||||
|
||||
```json
|
||||
{
|
||||
"maxConn": 100, // KM对单台Broker的最大JMX连接数
|
||||
"username": "xxxxx", //用户名，可以不填写
|
||||
"password": "xxxx", // 密码，可以不填写
|
||||
"openSSL": true, //开启SSL, true表示开启ssl, false表示关闭
|
||||
"useWhichEndpoint": "EXTERNAL" //指定要连接的网络名称，填写EXTERNAL就是连接endpoints里面的EXTERNAL地址
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
|
||||
SQL例子：
|
||||
|
||||
```sql
|
||||
UPDATE ks_km_physical_cluster SET jmx_properties='{ "maxConn": 10, "username": "xxxxx", "password": "xxxx", "openSSL": false , "useWhichEndpoint": "xxx"}' where id={xxx};
|
||||
```
|
||||
|
||||
### 2.2、正常情况
|
||||
|
||||
修改完成后，如果看到 JMX PORT这一列全部为绿色，则表示JMX已正常。
|
||||
|
||||
<img src=http://img-ys011.didistatic.com/static/dc2img/do1_ymtDTCiDlzfrmSCez2lx width="90%">
|
||||
|
||||
|
||||
|
||||
## 3、Elasticsearch问题
|
||||
|
||||
注意：mac系统在执行curl指令时，可能报zsh错误。可参考以下操作。
|
||||
|
||||
```
|
||||
1 进入.zshrc 文件 vim ~/.zshrc
|
||||
2.在.zshrc中加入 setopt no_nomatch
|
||||
3.更新配置 source ~/.zshrc
|
||||
```
|
||||
|
||||
### 3.1、原因一：缺少索引
|
||||
|
||||
#### 3.1.1、异常现象
|
||||
|
||||
报错信息
|
||||
|
||||
```
|
||||
com.didiglobal.logi.elasticsearch.client.model.exception.ESIndexNotFoundException: method [GET], host[http://127.0.0.1:9200], URI [/ks_kafka_broker_metric_2022-10-21,ks_kafka_broker_metric_2022-10-22/_search], status line [HTTP/1.1 404 Not Found]
|
||||
```
|
||||
|
||||
curl http://{ES的IP地址}:{ES的端口号}/_cat/indices/ks_kafka* 查看KS索引列表，发现没有索引。
|
||||
|
||||
#### 3.1.2、解决方案
|
||||
|
||||
执行[/km-dist/init/template/template.sh](https://github.com/didi/KnowStreaming/blob/master/km-dist/init/template/template.sh)脚本创建索引。
|
||||
|
||||
|
||||
|
||||
### 3.2、原因二：索引模板错误
|
||||
|
||||
#### 3.2.1、异常现象
|
||||
|
||||
多集群列表有数据，集群详情页图标无数据。查询KS索引模板列表，发现不存在。
|
||||
|
||||
```
|
||||
curl {ES的IP地址}:{ES的端口号}/_cat/templates/ks_kafka*?v&h=name
|
||||
```
|
||||
|
||||
正常KS模板如下图所示。
|
||||
|
||||
<img src=http://img-ys011.didistatic.com/static/dc2img/do1_l79bPYSci9wr6KFwZDA6 width="90%">
|
||||
|
||||
|
||||
|
||||
#### 3.2.2、解决方案
|
||||
|
||||
删除KS索引模板和索引
|
||||
|
||||
```
|
||||
curl -XDELETE {ES的IP地址}:{ES的端口号}/ks_kafka*
|
||||
curl -XDELETE {ES的IP地址}:{ES的端口号}/_template/ks_kafka*
|
||||
```
|
||||
|
||||
执行[/km-dist/init/template/template.sh](https://github.com/didi/KnowStreaming/blob/master/km-dist/init/template/template.sh)脚本初始化索引和模板。
|
||||
|
||||
|
||||
|
||||
### 3.3、原因三：集群Shard满
|
||||
|
||||
#### 3.3.1、异常现象
|
||||
|
||||
报错信息
|
||||
|
||||
```
|
||||
com.didiglobal.logi.elasticsearch.client.model.exception.ESIndexNotFoundException: method [GET], host[http://127.0.0.1:9200], URI [/ks_kafka_broker_metric_2022-10-21,ks_kafka_broker_metric_2022-10-22/_search], status line [HTTP/1.1 404 Not Found]
|
||||
```
|
||||
|
||||
尝试手动创建索引失败。
|
||||
|
||||
```
|
||||
#创建ks_kafka_cluster_metric_test索引的指令
|
||||
curl -s -XPUT http://{ES的IP地址}:{ES的端口号}/ks_kafka_cluster_metric_test
|
||||
```
|
||||
|
||||
#### 3.3.2、解决方案
|
||||
|
||||
ES索引的默认分片数量为1000，达到数量以后，索引创建失败。
|
||||
|
||||
+ 扩大ES索引数量上限，执行指令
|
||||
|
||||
```
|
||||
curl -XPUT -H"content-type:application/json" http://{ES的IP地址}:{ES的端口号}/_cluster/settings -d '
|
||||
{
|
||||
"persistent": {
|
||||
"cluster": {
|
||||
"max_shards_per_node":{索引上限，默认为1000}
|
||||
}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
执行[/km-dist/init/template/template.sh](https://github.com/didi/KnowStreaming/blob/master/km-dist/init/template/template.sh)脚本补全索引。
|
||||
@@ -4,269 +4,13 @@
|
||||
- 如果想升级至具体版本，需要将你当前版本至你期望使用版本的变更统统执行一遍，然后才能正常使用。
|
||||
- 如果中间某个版本没有升级信息，则表示该版本直接替换安装包即可从前一个版本升级至当前版本。
|
||||
|
||||
### 升级至 `master` 版本
|
||||
|
||||
### 6.2.0、升级至 `master` 版本
|
||||
|
||||
暂无
|
||||
|
||||
### 升级至 `3.2.0` 版本
|
||||
|
||||
**配置变更**
|
||||
|
||||
```yaml
|
||||
# 新增如下配置
|
||||
|
||||
spring:
|
||||
logi-job: # know-streaming 依赖的 logi-job 模块的数据库的配置，默认与 know-streaming 的数据库配置保持一致即可
|
||||
enable: true # true表示开启job任务, false表关闭。KS在部署上可以考虑部署两套服务，一套处理前端请求，一套执行job任务，此时可以通过该字段进行控制
|
||||
|
||||
# 线程池大小相关配置
|
||||
thread-pool:
|
||||
es:
|
||||
search: # es查询线程池
|
||||
thread-num: 20 # 线程池大小
|
||||
queue-size: 10000 # 队列大小
|
||||
|
||||
# 客户端池大小相关配置
|
||||
client-pool:
|
||||
kafka-admin:
|
||||
client-cnt: 1 # 每个Kafka集群创建的KafkaAdminClient数
|
||||
|
||||
# ES客户端配置
|
||||
es:
|
||||
index:
|
||||
expire: 15 # 索引过期天数，15表示超过15天的索引会被KS过期删除
|
||||
```
|
||||
|
||||
**SQL 变更**
|
||||
```sql
|
||||
DROP TABLE IF EXISTS `ks_kc_connect_cluster`;
|
||||
CREATE TABLE `ks_kc_connect_cluster` (
|
||||
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'Connect集群ID',
|
||||
`kafka_cluster_phy_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT 'Kafka集群ID',
|
||||
`name` varchar(128) NOT NULL DEFAULT '' COMMENT '集群名称',
|
||||
`group_name` varchar(128) NOT NULL DEFAULT '' COMMENT '集群Group名称',
|
||||
`cluster_url` varchar(1024) NOT NULL DEFAULT '' COMMENT '集群地址',
|
||||
`member_leader_url` varchar(1024) NOT NULL DEFAULT '' COMMENT 'URL地址',
|
||||
`version` varchar(64) NOT NULL DEFAULT '' COMMENT 'connect版本',
|
||||
`jmx_properties` text COMMENT 'JMX配置',
|
||||
`state` tinyint(4) NOT NULL DEFAULT '1' COMMENT '集群使用的消费组状态，也表示集群状态:-1 Unknown,0 ReBalance,1 Active,2 Dead,3 Empty',
|
||||
`create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '接入时间',
|
||||
`update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
|
||||
PRIMARY KEY (`id`),
|
||||
UNIQUE KEY `uniq_id_group_name` (`id`,`group_name`),
|
||||
UNIQUE KEY `uniq_name_kafka_cluster` (`name`,`kafka_cluster_phy_id`),
|
||||
KEY `idx_kafka_cluster_phy_id` (`kafka_cluster_phy_id`)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Connect集群信息表';
|
||||
|
||||
|
||||
DROP TABLE IF EXISTS `ks_kc_connector`;
|
||||
CREATE TABLE `ks_kc_connector` (
|
||||
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
|
||||
`kafka_cluster_phy_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT 'Kafka集群ID',
|
||||
`connect_cluster_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT 'Connect集群ID',
|
||||
`connector_name` varchar(512) NOT NULL DEFAULT '' COMMENT 'Connector名称',
|
||||
`connector_class_name` varchar(512) NOT NULL DEFAULT '' COMMENT 'Connector类',
|
||||
`connector_type` varchar(32) NOT NULL DEFAULT '' COMMENT 'Connector类型',
|
||||
`state` varchar(45) NOT NULL DEFAULT '' COMMENT '状态',
|
||||
`topics` text COMMENT '访问过的Topics',
|
||||
`task_count` int(11) NOT NULL DEFAULT '0' COMMENT '任务数',
|
||||
`create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
|
||||
`update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
|
||||
PRIMARY KEY (`id`),
|
||||
UNIQUE KEY `uniq_connect_cluster_id_connector_name` (`connect_cluster_id`,`connector_name`)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Connector信息表';
|
||||
|
||||
|
||||
DROP TABLE IF EXISTS `ks_kc_worker`;
|
||||
CREATE TABLE `ks_kc_worker` (
|
||||
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
|
||||
`kafka_cluster_phy_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT 'Kafka集群ID',
|
||||
`connect_cluster_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT 'Connect集群ID',
|
||||
`member_id` varchar(512) NOT NULL DEFAULT '' COMMENT '成员ID',
|
||||
`host` varchar(128) NOT NULL DEFAULT '' COMMENT '主机名',
|
||||
`jmx_port` int(16) NOT NULL DEFAULT '-1' COMMENT 'Jmx端口',
|
||||
`url` varchar(1024) NOT NULL DEFAULT '' COMMENT 'URL信息',
|
||||
`leader_url` varchar(1024) NOT NULL DEFAULT '' COMMENT 'leaderURL信息',
|
||||
`leader` int(16) NOT NULL DEFAULT '0' COMMENT '状态: 1是leader，0不是leader',
|
||||
`worker_id` varchar(128) NOT NULL COMMENT 'worker地址',
|
||||
`create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
|
||||
`update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
|
||||
PRIMARY KEY (`id`),
|
||||
UNIQUE KEY `uniq_cluster_id_member_id` (`connect_cluster_id`,`member_id`)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='worker信息表';
|
||||
|
||||
|
||||
DROP TABLE IF EXISTS `ks_kc_worker_connector`;
|
||||
CREATE TABLE `ks_kc_worker_connector` (
|
||||
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
|
||||
`kafka_cluster_phy_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT 'Kafka集群ID',
|
||||
`connect_cluster_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT 'Connect集群ID',
|
||||
`connector_name` varchar(512) NOT NULL DEFAULT '' COMMENT 'Connector名称',
|
||||
`worker_member_id` varchar(256) NOT NULL DEFAULT '',
|
||||
`task_id` int(16) NOT NULL DEFAULT '-1' COMMENT 'Task的ID',
|
||||
`state` varchar(128) DEFAULT NULL COMMENT '任务状态',
|
||||
`worker_id` varchar(128) DEFAULT NULL COMMENT 'worker信息',
|
||||
`create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
|
||||
`update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
|
||||
PRIMARY KEY (`id`),
|
||||
UNIQUE KEY `uniq_relation` (`connect_cluster_id`,`connector_name`,`task_id`,`worker_member_id`)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Worker和Connector关系表';
|
||||
|
||||
INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_CONNECTOR_FAILED_TASK_COUNT', '{\"value\" : 1}', 'connector失败状态的任务数量', 'admin');
|
||||
INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_CONNECTOR_UNASSIGNED_TASK_COUNT', '{\"value\" : 1}', 'connector未被分配的任务数量', 'admin');
|
||||
INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_CONNECT_CLUSTER_TASK_STARTUP_FAILURE_PERCENTAGE', '{\"value\" : 0.05}', 'Connect集群任务启动失败概率', 'admin');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 升级至 `v3.1.0` 版本
|
||||
|
||||
```sql
|
||||
INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_ZK_BRAIN_SPLIT', '{ \"value\": 1} ', 'ZK 脑裂', 'admin');
|
||||
INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_ZK_OUTSTANDING_REQUESTS', '{ \"amount\": 100, \"ratio\":0.8} ', 'ZK Outstanding 请求堆积数', 'admin');
|
||||
INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_ZK_WATCH_COUNT', '{ \"amount\": 100000, \"ratio\": 0.8 } ', 'ZK WatchCount 数', 'admin');
|
||||
INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_ZK_ALIVE_CONNECTIONS', '{ \"amount\": 10000, \"ratio\": 0.8 } ', 'ZK 连接数', 'admin');
|
||||
INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_ZK_APPROXIMATE_DATA_SIZE', '{ \"amount\": 524288000, \"ratio\": 0.8 } ', 'ZK 数据大小(Byte)', 'admin');
|
||||
INSERT INTO `ks_km_platform_cluster_config` (`cluster_id`, `value_group`, `value_name`, `value`, `description`, `operator`) VALUES ('-1', 'HEALTH', 'HC_ZK_SENT_RATE', '{ \"amount\": 500000, \"ratio\": 0.8 } ', 'ZK 发包数', 'admin');
|
||||
|
||||
```
|
||||
|
||||
### 升级至 `v3.0.1` 版本
|
||||
|
||||
**ES 索引模版**
|
||||
```bash
|
||||
# 新增 ks_kafka_zookeeper_metric 索引模版。
|
||||
# 可通过再次执行 bin/init_es_template.sh 脚本，创建该索引模版。
|
||||
|
||||
# 索引模版内容
|
||||
PUT _template/ks_kafka_zookeeper_metric
|
||||
{
|
||||
"order" : 10,
|
||||
"index_patterns" : [
|
||||
"ks_kafka_zookeeper_metric*"
|
||||
],
|
||||
"settings" : {
|
||||
"index" : {
|
||||
"number_of_shards" : "10"
|
||||
}
|
||||
},
|
||||
"mappings" : {
|
||||
"properties" : {
|
||||
"routingValue" : {
|
||||
"type" : "text",
|
||||
"fields" : {
|
||||
"keyword" : {
|
||||
"ignore_above" : 256,
|
||||
"type" : "keyword"
|
||||
}
|
||||
}
|
||||
},
|
||||
"clusterPhyId" : {
|
||||
"type" : "long"
|
||||
},
|
||||
"metrics" : {
|
||||
"properties" : {
|
||||
"AvgRequestLatency" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"MinRequestLatency" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"MaxRequestLatency" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"OutstandingRequests" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"NodeCount" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"WatchCount" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"NumAliveConnections" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"PacketsReceived" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"PacketsSent" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"EphemeralsCount" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"ApproximateDataSize" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"OpenFileDescriptorCount" : {
|
||||
"type" : "double"
|
||||
},
|
||||
"MaxFileDescriptorCount" : {
|
||||
"type" : "double"
|
||||
}
|
||||
}
|
||||
},
|
||||
"key" : {
|
||||
"type" : "text",
|
||||
"fields" : {
|
||||
"keyword" : {
|
||||
"ignore_above" : 256,
|
||||
"type" : "keyword"
|
||||
}
|
||||
}
|
||||
},
|
||||
"timestamp" : {
|
||||
"format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
|
||||
"type" : "date"
|
||||
}
|
||||
}
|
||||
},
|
||||
"aliases" : { }
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
**SQL 变更**
|
||||
|
||||
```sql
|
||||
DROP TABLE IF EXISTS `ks_km_zookeeper`;
|
||||
CREATE TABLE `ks_km_zookeeper` (
|
||||
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
|
||||
`cluster_phy_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT '物理集群ID',
|
||||
`host` varchar(128) NOT NULL DEFAULT '' COMMENT 'zookeeper主机名',
|
||||
`port` int(16) NOT NULL DEFAULT '-1' COMMENT 'zookeeper端口',
|
||||
`role` varchar(16) NOT NULL DEFAULT '' COMMENT '角色, leader follower observer',
|
||||
`version` varchar(128) NOT NULL DEFAULT '' COMMENT 'zookeeper版本',
|
||||
`status` int(16) NOT NULL DEFAULT '0' COMMENT '状态: 1存活，0未存活，11存活但是4字命令使用不了',
|
||||
`create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
|
||||
`update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
|
||||
PRIMARY KEY (`id`),
|
||||
UNIQUE KEY `uniq_cluster_phy_id_host_port` (`cluster_phy_id`,`host`, `port`)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Zookeeper信息表';
|
||||
|
||||
|
||||
DROP TABLE IF EXISTS `ks_km_group`;
|
||||
CREATE TABLE `ks_km_group` (
|
||||
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
|
||||
`cluster_phy_id` bigint(20) NOT NULL DEFAULT '-1' COMMENT '集群id',
|
||||
`name` varchar(192) COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT 'Group名称',
|
||||
`member_count` int(11) unsigned NOT NULL DEFAULT '0' COMMENT '成员数',
|
||||
`topic_members` text CHARACTER SET utf8 COMMENT 'group消费的topic列表',
|
||||
`partition_assignor` varchar(255) CHARACTER SET utf8 NOT NULL COMMENT '分配策略',
|
||||
`coordinator_id` int(11) NOT NULL COMMENT 'group协调器brokerId',
|
||||
`type` int(11) NOT NULL COMMENT 'group类型 0：consumer 1：connector',
|
||||
`state` varchar(64) CHARACTER SET utf8 NOT NULL DEFAULT '' COMMENT '状态',
|
||||
`create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
|
||||
`update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
|
||||
PRIMARY KEY (`id`),
|
||||
UNIQUE KEY `uniq_cluster_phy_id_name` (`cluster_phy_id`,`name`)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Group信息表';
|
||||
|
||||
```
|
||||
|
||||
|
||||
### 升级至 `v3.0.0` 版本
|
||||
### 6.2.1、升级至 `v3.0.0` 版本
|
||||
|
||||
**SQL 变更**
|
||||
|
||||
@@ -278,7 +22,7 @@ ADD COLUMN `zk_properties` TEXT NULL COMMENT 'ZK配置' AFTER `jmx_properties`;
|
||||
---
|
||||
|
||||
|
||||
### 升级至 `v3.0.0-beta.2`版本
|
||||
### 6.2.2、升级至 `v3.0.0-beta.2`版本
|
||||
|
||||
**配置变更**
|
||||
|
||||
@@ -349,7 +93,7 @@ ALTER TABLE `logi_security_oplog`
|
||||
|
||||
---
|
||||
|
||||
### 升级至 `v3.0.0-beta.1`版本
|
||||
### 6.2.3、升级至 `v3.0.0-beta.1`版本
|
||||
|
||||
**SQL 变更**
|
||||
|
||||
@@ -368,7 +112,7 @@ ALTER COLUMN `operation_methods` set default '';
|
||||
|
||||
---
|
||||
|
||||
### `2.x`版本 升级至 `v3.0.0-beta.0`版本
|
||||
### 6.2.4、`2.x`版本 升级至 `v3.0.0-beta.0`版本
|
||||
|
||||
**升级步骤：**
|
||||
|
||||
|
||||
@@ -37,7 +37,7 @@
|
||||
|
||||
## 8.4、`Jmx`连接失败如何解决？
|
||||
|
||||
- 参看 [Jmx 连接配置&问题解决](https://doc.knowstreaming.com/product/9-attachment#91jmx-%E8%BF%9E%E6%8E%A5%E5%A4%B1%E8%B4%A5%E9%97%AE%E9%A2%98%E8%A7%A3%E5%86%B3) 说明。
|
||||
- 参看 [Jmx 连接配置&问题解决](./9-attachment#jmx-连接失败问题解决) 说明。
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,15 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.biz.cluster;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.ClusterConnectorsOverviewDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.connect.ConnectStateVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.connector.ClusterConnectorOverviewVO;
|
||||
|
||||
/**
|
||||
* Kafka集群Connector概览
|
||||
*/
|
||||
public interface ClusterConnectorsManager {
|
||||
PaginationResult<ClusterConnectorOverviewVO> getClusterConnectorsOverview(Long clusterPhyId, ClusterConnectorsOverviewDTO dto);
|
||||
|
||||
ConnectStateVO getClusterConnectorsState(Long clusterPhyId);
|
||||
}
|
||||
@@ -1,19 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.biz.cluster;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.ClusterZookeepersOverviewDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.zookeeper.ClusterZookeepersOverviewVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.zookeeper.ClusterZookeepersStateVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.zookeeper.ZnodeVO;
|
||||
|
||||
/**
|
||||
* 多集群总体状态
|
||||
*/
|
||||
public interface ClusterZookeepersManager {
|
||||
Result<ClusterZookeepersStateVO> getClusterPhyZookeepersState(Long clusterPhyId);
|
||||
|
||||
PaginationResult<ClusterZookeepersOverviewVO> getClusterPhyZookeepersOverview(Long clusterPhyId, ClusterZookeepersOverviewDTO dto);
|
||||
|
||||
Result<ZnodeVO> getZnodeVO(Long clusterPhyId, String path);
|
||||
}
|
||||
@@ -1,6 +1,5 @@
|
||||
package com.xiaojukeji.know.streaming.km.biz.cluster;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhysHealthState;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhysState;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.MultiClusterDashboardDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
|
||||
@@ -16,8 +15,6 @@ public interface MultiClusterPhyManager {
|
||||
*/
|
||||
ClusterPhysState getClusterPhysState();
|
||||
|
||||
ClusterPhysHealthState getClusterPhysHealthState();
|
||||
|
||||
/**
|
||||
* 查询多集群大盘
|
||||
* @param dto 分页信息
|
||||
|
||||
@@ -6,8 +6,6 @@ import com.xiaojukeji.know.streaming.km.biz.cluster.ClusterBrokersManager;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.ClusterBrokersOverviewDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationSortDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.broker.Broker;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.config.JmxConfig;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.kafkacontroller.KafkaController;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.BrokerMetrics;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
|
||||
@@ -18,8 +16,6 @@ import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.res.ClusterBroker
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.kafkacontroller.KafkaControllerVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.KafkaConstant;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.SortTypeEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.cluster.ClusterRunStateEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.PaginationMetricsUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.PaginationUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
|
||||
@@ -28,8 +24,6 @@ import com.xiaojukeji.know.streaming.km.core.service.broker.BrokerMetricService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.broker.BrokerService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.kafkacontroller.KafkaControllerService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.topic.TopicService;
|
||||
import com.xiaojukeji.know.streaming.km.persistence.cache.LoadedClusterPhyCache;
|
||||
import com.xiaojukeji.know.streaming.km.persistence.kafka.KafkaJMXClient;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
@@ -57,9 +51,6 @@ public class ClusterBrokersManagerImpl implements ClusterBrokersManager {
|
||||
@Autowired
|
||||
private KafkaControllerService kafkaControllerService;
|
||||
|
||||
@Autowired
|
||||
private KafkaJMXClient kafkaJMXClient;
|
||||
|
||||
@Override
|
||||
public PaginationResult<ClusterBrokersOverviewVO> getClusterPhyBrokersOverview(Long clusterPhyId, ClusterBrokersOverviewDTO dto) {
|
||||
// 获取集群Broker列表
|
||||
@@ -84,24 +75,15 @@ public class ClusterBrokersManagerImpl implements ClusterBrokersManager {
|
||||
//获取controller信息
|
||||
KafkaController kafkaController = kafkaControllerService.getKafkaControllerFromDB(clusterPhyId);
|
||||
|
||||
//获取jmx状态信息
|
||||
Map<Integer, Boolean> jmxConnectedMap = new HashMap<>();
|
||||
brokerList.forEach(elem -> jmxConnectedMap.put(elem.getBrokerId(), kafkaJMXClient.getClientWithCheck(clusterPhyId, elem.getBrokerId()) != null));
|
||||
|
||||
|
||||
ClusterPhy clusterPhy = LoadedClusterPhyCache.getByPhyId(clusterPhyId);
|
||||
|
||||
// 格式转换
|
||||
return PaginationResult.buildSuc(
|
||||
this.convert2ClusterBrokersOverviewVOList(
|
||||
clusterPhy,
|
||||
paginationResult.getData().getBizData(),
|
||||
brokerList,
|
||||
metricsResult.getData(),
|
||||
groupTopic,
|
||||
transactionTopic,
|
||||
kafkaController,
|
||||
jmxConnectedMap
|
||||
kafkaController
|
||||
),
|
||||
paginationResult
|
||||
);
|
||||
@@ -178,36 +160,27 @@ public class ClusterBrokersManagerImpl implements ClusterBrokersManager {
|
||||
);
|
||||
}
|
||||
|
||||
private List<ClusterBrokersOverviewVO> convert2ClusterBrokersOverviewVOList(ClusterPhy clusterPhy,
|
||||
List<Integer> pagedBrokerIdList,
|
||||
private List<ClusterBrokersOverviewVO> convert2ClusterBrokersOverviewVOList(List<Integer> pagedBrokerIdList,
|
||||
List<Broker> brokerList,
|
||||
List<BrokerMetrics> metricsList,
|
||||
Topic groupTopic,
|
||||
Topic transactionTopic,
|
||||
KafkaController kafkaController,
|
||||
Map<Integer, Boolean> jmxConnectedMap) {
|
||||
Map<Integer, BrokerMetrics> metricsMap = metricsList == null ? new HashMap<>() : metricsList.stream().collect(Collectors.toMap(BrokerMetrics::getBrokerId, Function.identity()));
|
||||
KafkaController kafkaController) {
|
||||
Map<Integer, BrokerMetrics> metricsMap = metricsList == null? new HashMap<>(): metricsList.stream().collect(Collectors.toMap(BrokerMetrics::getBrokerId, Function.identity()));
|
||||
|
||||
Map<Integer, Broker> brokerMap = brokerList == null ? new HashMap<>() : brokerList.stream().collect(Collectors.toMap(Broker::getBrokerId, Function.identity()));
|
||||
Map<Integer, Broker> brokerMap = brokerList == null? new HashMap<>(): brokerList.stream().collect(Collectors.toMap(Broker::getBrokerId, Function.identity()));
|
||||
|
||||
List<ClusterBrokersOverviewVO> voList = new ArrayList<>(pagedBrokerIdList.size());
|
||||
for (Integer brokerId : pagedBrokerIdList) {
|
||||
Broker broker = brokerMap.get(brokerId);
|
||||
BrokerMetrics brokerMetrics = metricsMap.get(brokerId);
|
||||
Boolean jmxConnected = jmxConnectedMap.get(brokerId);
|
||||
voList.add(this.convert2ClusterBrokersOverviewVO(brokerId, broker, brokerMetrics, groupTopic, transactionTopic, kafkaController, jmxConnected));
|
||||
}
|
||||
|
||||
//补充非zk模式的JMXPort信息
|
||||
if (!clusterPhy.getRunState().equals(ClusterRunStateEnum.RUN_ZK.getRunState())) {
|
||||
JmxConfig jmxConfig = ConvertUtil.str2ObjByJson(clusterPhy.getJmxProperties(), JmxConfig.class);
|
||||
voList.forEach(elem -> elem.setJmxPort(jmxConfig.getJmxPort() == null ? -1 : jmxConfig.getJmxPort()));
|
||||
voList.add(this.convert2ClusterBrokersOverviewVO(brokerId, broker, brokerMetrics, groupTopic, transactionTopic, kafkaController));
|
||||
}
|
||||
|
||||
return voList;
|
||||
}
|
||||
|
||||
private ClusterBrokersOverviewVO convert2ClusterBrokersOverviewVO(Integer brokerId, Broker broker, BrokerMetrics brokerMetrics, Topic groupTopic, Topic transactionTopic, KafkaController kafkaController, Boolean jmxConnected) {
|
||||
private ClusterBrokersOverviewVO convert2ClusterBrokersOverviewVO(Integer brokerId, Broker broker, BrokerMetrics brokerMetrics, Topic groupTopic, Topic transactionTopic, KafkaController kafkaController) {
|
||||
ClusterBrokersOverviewVO clusterBrokersOverviewVO = new ClusterBrokersOverviewVO();
|
||||
clusterBrokersOverviewVO.setBrokerId(brokerId);
|
||||
if (broker != null) {
|
||||
@@ -230,7 +203,6 @@ public class ClusterBrokersManagerImpl implements ClusterBrokersManager {
|
||||
}
|
||||
|
||||
clusterBrokersOverviewVO.setLatestMetrics(brokerMetrics);
|
||||
clusterBrokersOverviewVO.setJmxConnected(jmxConnected);
|
||||
return clusterBrokersOverviewVO;
|
||||
}
|
||||
|
||||
|
||||
@@ -1,152 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.biz.cluster.impl;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.biz.cluster.ClusterConnectorsManager;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.ClusterConnectorsOverviewDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.ClusterConnectorDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.MetricDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.connect.MetricsConnectorsDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.ConnectCluster;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.ConnectWorker;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.WorkerConnector;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.connect.ConnectorMetrics;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.po.connect.ConnectorPO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.connect.ConnectStateVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.connector.ClusterConnectorOverviewVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.metrics.line.MetricMultiLinesVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.converter.ConnectConverter;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.PaginationMetricsUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.PaginationUtil;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.connect.cluster.ConnectClusterService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.connect.connector.ConnectorMetricService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.connect.connector.ConnectorService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.connect.worker.WorkerConnectorService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.connect.worker.WorkerService;
|
||||
import org.apache.kafka.connect.runtime.AbstractStatus;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
|
||||
@Service
|
||||
public class ClusterConnectorsManagerImpl implements ClusterConnectorsManager {
|
||||
private static final ILog LOGGER = LogFactory.getLog(ClusterConnectorsManagerImpl.class);
|
||||
|
||||
@Autowired
|
||||
private ConnectorService connectorService;
|
||||
|
||||
@Autowired
|
||||
private ConnectClusterService connectClusterService;
|
||||
|
||||
@Autowired
|
||||
private ConnectorMetricService connectorMetricService;
|
||||
|
||||
@Autowired
|
||||
private WorkerService workerService;
|
||||
|
||||
@Autowired
|
||||
private WorkerConnectorService workerConnectorService;
|
||||
|
||||
@Override
|
||||
public PaginationResult<ClusterConnectorOverviewVO> getClusterConnectorsOverview(Long clusterPhyId, ClusterConnectorsOverviewDTO dto) {
|
||||
List<ConnectCluster> clusterList = connectClusterService.listByKafkaCluster(clusterPhyId);
|
||||
|
||||
List<ConnectorPO> poList = connectorService.listByKafkaClusterIdFromDB(clusterPhyId);
|
||||
|
||||
// 查询实时指标
|
||||
Result<List<ConnectorMetrics>> latestMetricsResult = connectorMetricService.getLatestMetricsFromES(
|
||||
clusterPhyId,
|
||||
poList.stream().map(elem -> new ClusterConnectorDTO(elem.getConnectClusterId(), elem.getConnectorName())).collect(Collectors.toList()),
|
||||
dto.getLatestMetricNames()
|
||||
);
|
||||
|
||||
if (latestMetricsResult.failed()) {
|
||||
LOGGER.error("method=getClusterConnectorsOverview||clusterPhyId={}||result={}||errMsg=get latest metric failed", clusterPhyId, latestMetricsResult);
|
||||
return PaginationResult.buildFailure(latestMetricsResult, dto);
|
||||
}
|
||||
|
||||
// 转换成vo
|
||||
List<ClusterConnectorOverviewVO> voList = ConnectConverter.convert2ClusterConnectorOverviewVOList(clusterList, poList,latestMetricsResult.getData());
|
||||
|
||||
// 请求分页信息
|
||||
PaginationResult<ClusterConnectorOverviewVO> voPaginationResult = this.pagingConnectorInLocal(voList, dto);
|
||||
if (voPaginationResult.failed()) {
|
||||
LOGGER.error("method=getClusterConnectorsOverview||clusterPhyId={}||result={}||errMsg=pagination in local failed", clusterPhyId, voPaginationResult);
|
||||
|
||||
return PaginationResult.buildFailure(voPaginationResult, dto);
|
||||
}
|
||||
|
||||
// 查询历史指标
|
||||
Result<List<MetricMultiLinesVO>> lineMetricsResult = connectorMetricService.listConnectClusterMetricsFromES(
|
||||
clusterPhyId,
|
||||
this.buildMetricsConnectorsDTO(
|
||||
voPaginationResult.getData().getBizData().stream().map(elem -> new ClusterConnectorDTO(elem.getConnectClusterId(), elem.getConnectorName())).collect(Collectors.toList()),
|
||||
dto.getMetricLines()
|
||||
)
|
||||
);
|
||||
|
||||
|
||||
return PaginationResult.buildSuc(
|
||||
ConnectConverter.supplyData2ClusterConnectorOverviewVOList(
|
||||
voPaginationResult.getData().getBizData(),
|
||||
lineMetricsResult.getData()
|
||||
),
|
||||
voPaginationResult
|
||||
);
|
||||
}
|
||||
|
||||
@Override
|
||||
public ConnectStateVO getClusterConnectorsState(Long clusterPhyId) {
|
||||
//获取Connect集群Id列表
|
||||
List<ConnectCluster> connectClusterList = connectClusterService.listByKafkaCluster(clusterPhyId);
|
||||
List<ConnectorPO> connectorPOList = connectorService.listByKafkaClusterIdFromDB(clusterPhyId);
|
||||
List<WorkerConnector> workerConnectorList = workerConnectorService.listByKafkaClusterIdFromDB(clusterPhyId);
|
||||
List<ConnectWorker> connectWorkerList = workerService.listByKafkaClusterIdFromDB(clusterPhyId);
|
||||
|
||||
return convert2ConnectStateVO(connectClusterList, connectorPOList, workerConnectorList, connectWorkerList);
|
||||
}
|
||||
|
||||
/**************************************************** private method ****************************************************/
|
||||
|
||||
private MetricsConnectorsDTO buildMetricsConnectorsDTO(List<ClusterConnectorDTO> connectorDTOList, MetricDTO metricDTO) {
|
||||
MetricsConnectorsDTO dto = ConvertUtil.obj2Obj(metricDTO, MetricsConnectorsDTO.class);
|
||||
dto.setConnectorNameList(connectorDTOList == null? new ArrayList<>(): connectorDTOList);
|
||||
|
||||
return dto;
|
||||
}
|
||||
|
||||
private ConnectStateVO convert2ConnectStateVO(List<ConnectCluster> connectClusterList, List<ConnectorPO> connectorPOList, List<WorkerConnector> workerConnectorList, List<ConnectWorker> connectWorkerList) {
|
||||
ConnectStateVO connectStateVO = new ConnectStateVO();
|
||||
connectStateVO.setConnectClusterCount(connectClusterList.size());
|
||||
connectStateVO.setTotalConnectorCount(connectorPOList.size());
|
||||
connectStateVO.setAliveConnectorCount(connectorPOList.stream().filter(elem -> elem.getState().equals(AbstractStatus.State.RUNNING.name())).collect(Collectors.toList()).size());
|
||||
connectStateVO.setWorkerCount(connectWorkerList.size());
|
||||
connectStateVO.setTotalTaskCount(workerConnectorList.size());
|
||||
connectStateVO.setAliveTaskCount(workerConnectorList.stream().filter(elem -> elem.getState().equals(AbstractStatus.State.RUNNING.name())).collect(Collectors.toList()).size());
|
||||
return connectStateVO;
|
||||
}
|
||||
|
||||
private PaginationResult<ClusterConnectorOverviewVO> pagingConnectorInLocal(List<ClusterConnectorOverviewVO> connectorVOList, ClusterConnectorsOverviewDTO dto) {
|
||||
//模糊匹配
|
||||
connectorVOList = PaginationUtil.pageByFuzzyFilter(connectorVOList, dto.getSearchKeywords(), Arrays.asList("connectClusterName"));
|
||||
|
||||
//排序
|
||||
if (!dto.getLatestMetricNames().isEmpty()) {
|
||||
PaginationMetricsUtil.sortMetrics(connectorVOList, "latestMetrics", dto.getSortMetricNameList(), "connectClusterName", dto.getSortType());
|
||||
} else {
|
||||
PaginationUtil.pageBySort(connectorVOList, dto.getSortField(), dto.getSortType(), "connectClusterName", dto.getSortType());
|
||||
}
|
||||
|
||||
//分页
|
||||
return PaginationUtil.pageBySubData(connectorVOList, dto);
|
||||
}
|
||||
|
||||
}
|
||||
@@ -44,7 +44,7 @@ public class ClusterTopicsManagerImpl implements ClusterTopicsManager {
|
||||
List<Topic> topicList = topicService.listTopicsFromDB(clusterPhyId);
|
||||
|
||||
// 获取集群所有Topic的指标
|
||||
Map<String, TopicMetrics> metricsMap = topicMetricService.getLatestMetricsFromCache(clusterPhyId);
|
||||
Map<String, TopicMetrics> metricsMap = topicMetricService.getLatestMetricsFromCacheFirst(clusterPhyId);
|
||||
|
||||
// 转换成vo
|
||||
List<ClusterPhyTopicsOverviewVO> voList = TopicVOConverter.convert2ClusterPhyTopicsOverviewVOList(topicList, metricsMap);
|
||||
|
||||
@@ -1,137 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.biz.cluster.impl;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.biz.cluster.ClusterZookeepersManager;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.ClusterZookeepersOverviewDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.ZookeeperMetrics;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.ResultStatus;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.zookeeper.Znode;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.zookeeper.ZookeeperInfo;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.zookeeper.ClusterZookeepersOverviewVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.zookeeper.ClusterZookeepersStateVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.zookeeper.ZnodeVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.MsgConstant;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.zookeeper.ZKRoleEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.PaginationUtil;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterPhyService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.ZookeeperMetricVersionItems;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.zookeeper.ZnodeService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.zookeeper.ZookeeperMetricService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.zookeeper.ZookeeperService;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Service;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
|
||||
|
||||
@Service
|
||||
public class ClusterZookeepersManagerImpl implements ClusterZookeepersManager {
|
||||
private static final ILog LOGGER = LogFactory.getLog(ClusterZookeepersManagerImpl.class);
|
||||
|
||||
@Autowired
|
||||
private ClusterPhyService clusterPhyService;
|
||||
|
||||
@Autowired
|
||||
private ZookeeperService zookeeperService;
|
||||
|
||||
@Autowired
|
||||
private ZookeeperMetricService zookeeperMetricService;
|
||||
|
||||
@Autowired
|
||||
private ZnodeService znodeService;
|
||||
|
||||
@Override
|
||||
public Result<ClusterZookeepersStateVO> getClusterPhyZookeepersState(Long clusterPhyId) {
|
||||
ClusterPhy clusterPhy = clusterPhyService.getClusterByCluster(clusterPhyId);
|
||||
if (clusterPhy == null) {
|
||||
return Result.buildFromRSAndMsg(ResultStatus.CLUSTER_NOT_EXIST, MsgConstant.getClusterPhyNotExist(clusterPhyId));
|
||||
}
|
||||
|
||||
List<ZookeeperInfo> infoList = zookeeperService.listFromDBByCluster(clusterPhyId);
|
||||
|
||||
ClusterZookeepersStateVO vo = new ClusterZookeepersStateVO();
|
||||
vo.setTotalServerCount(infoList.size());
|
||||
vo.setAliveFollowerCount(0);
|
||||
vo.setTotalFollowerCount(0);
|
||||
vo.setAliveObserverCount(0);
|
||||
vo.setTotalObserverCount(0);
|
||||
vo.setAliveServerCount(0);
|
||||
for (ZookeeperInfo info: infoList) {
|
||||
if (info.getRole().equals(ZKRoleEnum.LEADER.getRole())) {
|
||||
vo.setLeaderNode(info.getHost());
|
||||
}
|
||||
|
||||
if (info.getRole().equals(ZKRoleEnum.FOLLOWER.getRole())) {
|
||||
vo.setTotalFollowerCount(vo.getTotalFollowerCount() + 1);
|
||||
vo.setAliveFollowerCount(info.alive()? vo.getAliveFollowerCount() + 1: vo.getAliveFollowerCount());
|
||||
}
|
||||
|
||||
if (info.getRole().equals(ZKRoleEnum.OBSERVER.getRole())) {
|
||||
vo.setTotalObserverCount(vo.getTotalObserverCount() + 1);
|
||||
vo.setAliveObserverCount(info.alive()? vo.getAliveObserverCount() + 1: vo.getAliveObserverCount());
|
||||
}
|
||||
|
||||
if (info.alive()) {
|
||||
vo.setAliveServerCount(vo.getAliveServerCount() + 1);
|
||||
}
|
||||
}
|
||||
|
||||
// 指标获取
|
||||
Result<ZookeeperMetrics> metricsResult = zookeeperMetricService.batchCollectMetricsFromZookeeper(
|
||||
clusterPhyId,
|
||||
Arrays.asList(
|
||||
ZookeeperMetricVersionItems.ZOOKEEPER_METRIC_WATCH_COUNT,
|
||||
ZookeeperMetricVersionItems.ZOOKEEPER_METRIC_HEALTH_STATE,
|
||||
ZookeeperMetricVersionItems.ZOOKEEPER_METRIC_HEALTH_CHECK_PASSED,
|
||||
ZookeeperMetricVersionItems.ZOOKEEPER_METRIC_HEALTH_CHECK_TOTAL
|
||||
)
|
||||
|
||||
);
|
||||
if (metricsResult.failed()) {
|
||||
LOGGER.error(
|
||||
"method=getClusterPhyZookeepersState||clusterPhyId={}||errMsg={}",
|
||||
clusterPhyId, metricsResult.getMessage()
|
||||
);
|
||||
return Result.buildSuc(vo);
|
||||
}
|
||||
|
||||
ZookeeperMetrics metrics = metricsResult.getData();
|
||||
vo.setWatchCount(ConvertUtil.float2Integer(metrics.getMetrics().get(ZookeeperMetricVersionItems.ZOOKEEPER_METRIC_WATCH_COUNT)));
|
||||
vo.setHealthState(ConvertUtil.float2Integer(metrics.getMetrics().get(ZookeeperMetricVersionItems.ZOOKEEPER_METRIC_HEALTH_STATE)));
|
||||
vo.setHealthCheckPassed(ConvertUtil.float2Integer(metrics.getMetrics().get(ZookeeperMetricVersionItems.ZOOKEEPER_METRIC_HEALTH_CHECK_PASSED)));
|
||||
vo.setHealthCheckTotal(ConvertUtil.float2Integer(metrics.getMetrics().get(ZookeeperMetricVersionItems.ZOOKEEPER_METRIC_HEALTH_CHECK_TOTAL)));
|
||||
|
||||
return Result.buildSuc(vo);
|
||||
}
|
||||
|
||||
@Override
|
||||
public PaginationResult<ClusterZookeepersOverviewVO> getClusterPhyZookeepersOverview(Long clusterPhyId, ClusterZookeepersOverviewDTO dto) {
|
||||
//获取集群zookeeper列表
|
||||
List<ClusterZookeepersOverviewVO> clusterZookeepersOverviewVOList = ConvertUtil.list2List(zookeeperService.listFromDBByCluster(clusterPhyId), ClusterZookeepersOverviewVO.class);
|
||||
|
||||
//搜索
|
||||
clusterZookeepersOverviewVOList = PaginationUtil.pageByFuzzyFilter(clusterZookeepersOverviewVOList, dto.getSearchKeywords(), Arrays.asList("host"));
|
||||
|
||||
//分页
|
||||
PaginationResult<ClusterZookeepersOverviewVO> paginationResult = PaginationUtil.pageBySubData(clusterZookeepersOverviewVOList, dto);
|
||||
|
||||
return paginationResult;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Result<ZnodeVO> getZnodeVO(Long clusterPhyId, String path) {
|
||||
Result<Znode> result = znodeService.getZnode(clusterPhyId, path);
|
||||
if (result.failed()) {
|
||||
return Result.buildFromIgnoreData(result);
|
||||
}
|
||||
return Result.buildSuc(ConvertUtil.obj2ObjByJSON(result.getData(), ZnodeVO.class));
|
||||
}
|
||||
|
||||
/**************************************************** private method ****************************************************/
|
||||
|
||||
}
|
||||
@@ -5,7 +5,6 @@ import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.biz.cluster.MultiClusterPhyManager;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.MetricDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.MetricsClusterPhyDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhysHealthState;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhysState;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.MultiClusterDashboardDTO;
|
||||
@@ -17,7 +16,6 @@ import com.xiaojukeji.know.streaming.km.common.bean.vo.cluster.ClusterPhyDashboa
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.metrics.line.MetricMultiLinesVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.Constant;
|
||||
import com.xiaojukeji.know.streaming.km.common.converter.ClusterVOConverter;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.health.HealthStateEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.PaginationUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.PaginationMetricsUtil;
|
||||
@@ -25,11 +23,14 @@ import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterMetricService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterPhyService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.kafkacontroller.KafkaControllerService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.ClusterMetricVersionItems;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.metrics.ClusterMetricVersionItems;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
@Service
|
||||
@@ -54,6 +55,7 @@ public class MultiClusterPhyManagerImpl implements MultiClusterPhyManager {
|
||||
false
|
||||
);
|
||||
|
||||
// TODO 后续产品上，看是否需要增加一个未知的状态，否则新接入的集群，因为新接入的集群，数据存在延迟
|
||||
ClusterPhysState physState = new ClusterPhysState(0, 0, clusterPhyList.size());
|
||||
for (ClusterPhy clusterPhy: clusterPhyList) {
|
||||
KafkaController kafkaController = controllerMap.get(clusterPhy.getId());
|
||||
@@ -73,32 +75,6 @@ public class MultiClusterPhyManagerImpl implements MultiClusterPhyManager {
|
||||
return physState;
|
||||
}
|
||||
|
||||
@Override
|
||||
public ClusterPhysHealthState getClusterPhysHealthState() {
|
||||
List<ClusterPhy> clusterPhyList = clusterPhyService.listAllClusters();
|
||||
|
||||
ClusterPhysHealthState physState = new ClusterPhysHealthState(clusterPhyList.size());
|
||||
for (ClusterPhy clusterPhy: clusterPhyList) {
|
||||
ClusterMetrics metrics = clusterMetricService.getLatestMetricsFromCache(clusterPhy.getId());
|
||||
Float state = metrics.getMetric(ClusterMetricVersionItems.CLUSTER_METRIC_HEALTH_STATE);
|
||||
if (state == null) {
|
||||
physState.setUnknownCount(physState.getUnknownCount() + 1);
|
||||
} else if (state.intValue() == HealthStateEnum.GOOD.getDimension()) {
|
||||
physState.setGoodCount(physState.getGoodCount() + 1);
|
||||
} else if (state.intValue() == HealthStateEnum.MEDIUM.getDimension()) {
|
||||
physState.setMediumCount(physState.getMediumCount() + 1);
|
||||
} else if (state.intValue() == HealthStateEnum.POOR.getDimension()) {
|
||||
physState.setPoorCount(physState.getPoorCount() + 1);
|
||||
} else if (state.intValue() == HealthStateEnum.DEAD.getDimension()) {
|
||||
physState.setDeadCount(physState.getDeadCount() + 1);
|
||||
} else {
|
||||
physState.setUnknownCount(physState.getUnknownCount() + 1);
|
||||
}
|
||||
}
|
||||
|
||||
return physState;
|
||||
}
|
||||
|
||||
@Override
|
||||
public PaginationResult<ClusterPhyDashboardVO> getClusterPhysDashboard(MultiClusterDashboardDTO dto) {
|
||||
// 获取集群
|
||||
@@ -107,6 +83,7 @@ public class MultiClusterPhyManagerImpl implements MultiClusterPhyManager {
|
||||
// 转为vo格式，方便后续进行分页筛选等
|
||||
List<ClusterPhyDashboardVO> voList = ConvertUtil.list2List(clusterPhyList, ClusterPhyDashboardVO.class);
|
||||
|
||||
// TODO 后续产品上，看是否需要增加一个未知的状态，否则新接入的集群，因为新接入的集群，数据存在延迟
|
||||
// 获取集群controller信息并补充到vo中,
|
||||
Map<Long, KafkaController> controllerMap = kafkaControllerService.getKafkaControllersFromDB(clusterPhyList.stream().map(elem -> elem.getId()).collect(Collectors.toList()), false);
|
||||
for (ClusterPhyDashboardVO vo: voList) {
|
||||
@@ -172,7 +149,13 @@ public class MultiClusterPhyManagerImpl implements MultiClusterPhyManager {
|
||||
List<ClusterMetrics> metricsList = new ArrayList<>();
|
||||
for (ClusterPhyDashboardVO vo: voList) {
|
||||
ClusterMetrics clusterMetrics = clusterMetricService.getLatestMetricsFromCache(vo.getId());
|
||||
clusterMetrics.getMetrics().putIfAbsent(ClusterMetricVersionItems.CLUSTER_METRIC_HEALTH_STATE, (float) HealthStateEnum.UNKNOWN.getDimension());
|
||||
if (!clusterMetrics.getMetrics().containsKey(ClusterMetricVersionItems.CLUSTER_METRIC_HEALTH_SCORE)) {
|
||||
Float alive = clusterMetrics.getMetrics().get(ClusterMetricVersionItems.CLUSTER_METRIC_ALIVE);
|
||||
// 如果集群没有健康分，则设置一个默认的健康分数值
|
||||
clusterMetrics.putMetric(ClusterMetricVersionItems.CLUSTER_METRIC_HEALTH_SCORE,
|
||||
(alive != null && alive <= 0)? 0.0f: Constant.DEFAULT_CLUSTER_HEALTH_SCORE.floatValue()
|
||||
);
|
||||
}
|
||||
|
||||
metricsList.add(clusterMetrics);
|
||||
}
|
||||
|
||||
@@ -1,15 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.biz.connect.connector;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector.ConnectorCreateDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.connect.connector.ConnectorStateVO;
|
||||
|
||||
import java.util.Properties;
|
||||
|
||||
public interface ConnectorManager {
|
||||
Result<Void> updateConnectorConfig(Long connectClusterId, String connectorName, Properties configs, String operator);
|
||||
|
||||
Result<Void> createConnector(ConnectorCreateDTO dto, String operator);
|
||||
|
||||
Result<ConnectorStateVO> getConnectorStateVO(Long connectClusterId, String connectorName);
|
||||
}
|
||||
@@ -1,16 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.biz.connect.connector;
|
||||
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.connect.task.KCTaskOverviewVO;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* @author wyb
|
||||
* @date 2022/11/14
|
||||
*/
|
||||
public interface WorkerConnectorManager {
|
||||
Result<List<KCTaskOverviewVO>> getTaskOverview(Long connectClusterId, String connectorName);
|
||||
|
||||
}
|
||||
@@ -1,93 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.biz.connect.connector.impl;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.biz.connect.connector.ConnectorManager;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector.ConnectorCreateDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.WorkerConnector;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.config.ConnectConfigInfos;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.connector.KSConnector;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.connector.KSConnectorInfo;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.ResultStatus;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.po.connect.ConnectorPO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.connect.connector.ConnectorStateVO;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.connect.connector.ConnectorService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.connect.plugin.PluginService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.connect.worker.WorkerConnectorService;
|
||||
import org.apache.kafka.connect.runtime.AbstractStatus;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Properties;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
@Service
|
||||
public class ConnectorManagerImpl implements ConnectorManager {
|
||||
private static final ILog LOGGER = LogFactory.getLog(ConnectorManagerImpl.class);
|
||||
|
||||
@Autowired
|
||||
private PluginService pluginService;
|
||||
|
||||
@Autowired
|
||||
private ConnectorService connectorService;
|
||||
|
||||
@Autowired
|
||||
private WorkerConnectorService workerConnectorService;
|
||||
|
||||
@Override
|
||||
public Result<Void> updateConnectorConfig(Long connectClusterId, String connectorName, Properties configs, String operator) {
|
||||
Result<ConnectConfigInfos> infosResult = pluginService.validateConfig(connectClusterId, configs);
|
||||
if (infosResult.failed()) {
|
||||
return Result.buildFromIgnoreData(infosResult);
|
||||
}
|
||||
|
||||
if (infosResult.getData().getErrorCount() > 0) {
|
||||
return Result.buildFromRSAndMsg(ResultStatus.PARAM_ILLEGAL, "Connector参数错误");
|
||||
}
|
||||
|
||||
return connectorService.updateConnectorConfig(connectClusterId, connectorName, configs, operator);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Result<Void> createConnector(ConnectorCreateDTO dto, String operator) {
|
||||
Result<KSConnectorInfo> createResult = connectorService.createConnector(dto.getConnectClusterId(), dto.getConnectorName(), dto.getConfigs(), operator);
|
||||
if (createResult.failed()) {
|
||||
return Result.buildFromIgnoreData(createResult);
|
||||
}
|
||||
|
||||
Result<KSConnector> ksConnectorResult = connectorService.getAllConnectorInfoFromCluster(dto.getConnectClusterId(), dto.getConnectorName());
|
||||
if (ksConnectorResult.failed()) {
|
||||
return Result.buildFromRSAndMsg(ResultStatus.SUCCESS, "创建成功，但是获取元信息失败，页面元信息会存在1分钟延迟");
|
||||
}
|
||||
|
||||
connectorService.addNewToDB(ksConnectorResult.getData());
|
||||
return Result.buildSuc();
|
||||
}
|
||||
|
||||
@Override
|
||||
public Result<ConnectorStateVO> getConnectorStateVO(Long connectClusterId, String connectorName) {
|
||||
ConnectorPO connectorPO = connectorService.getConnectorFromDB(connectClusterId, connectorName);
|
||||
|
||||
if (connectorPO == null) {
|
||||
return Result.buildFailure(ResultStatus.NOT_EXIST);
|
||||
}
|
||||
|
||||
List<WorkerConnector> workerConnectorList = workerConnectorService.listFromDB(connectClusterId).stream().filter(elem -> elem.getConnectorName().equals(connectorName)).collect(Collectors.toList());
|
||||
|
||||
return Result.buildSuc(convert2ConnectorOverviewVO(connectorPO, workerConnectorList));
|
||||
}
|
||||
|
||||
private ConnectorStateVO convert2ConnectorOverviewVO(ConnectorPO connectorPO, List<WorkerConnector> workerConnectorList) {
|
||||
ConnectorStateVO connectorStateVO = new ConnectorStateVO();
|
||||
connectorStateVO.setConnectClusterId(connectorPO.getConnectClusterId());
|
||||
connectorStateVO.setName(connectorPO.getConnectorName());
|
||||
connectorStateVO.setType(connectorPO.getConnectorType());
|
||||
connectorStateVO.setState(connectorPO.getState());
|
||||
connectorStateVO.setTotalTaskCount(workerConnectorList.size());
|
||||
connectorStateVO.setAliveTaskCount(workerConnectorList.stream().filter(elem -> elem.getState().equals(AbstractStatus.State.RUNNING.name())).collect(Collectors.toList()).size());
|
||||
connectorStateVO.setTotalWorkerCount(workerConnectorList.stream().map(elem -> elem.getWorkerId()).collect(Collectors.toSet()).size());
|
||||
return connectorStateVO;
|
||||
}
|
||||
}
|
||||
@@ -1,37 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.biz.connect.connector.impl;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.biz.connect.connector.WorkerConnectorManager;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.ConnectCluster;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.WorkerConnector;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.connect.task.KCTaskOverviewVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.connect.worker.WorkerConnectorService;
|
||||
import com.xiaojukeji.know.streaming.km.persistence.connect.cache.LoadedConnectClusterCache;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* @author wyb
|
||||
* @date 2022/11/14
|
||||
*/
|
||||
@Service
|
||||
public class WorkerConnectorManageImpl implements WorkerConnectorManager {
|
||||
|
||||
private static final ILog LOGGER = LogFactory.getLog(WorkerConnectorManageImpl.class);
|
||||
|
||||
@Autowired
|
||||
private WorkerConnectorService workerConnectorService;
|
||||
|
||||
@Override
|
||||
public Result<List<KCTaskOverviewVO>> getTaskOverview(Long connectClusterId, String connectorName) {
|
||||
ConnectCluster connectCluster = LoadedConnectClusterCache.getByPhyId(connectClusterId);
|
||||
List<WorkerConnector> workerConnectorList = workerConnectorService.getWorkerConnectorListFromCluster(connectCluster, connectorName);
|
||||
|
||||
return Result.buildSuc(ConvertUtil.list2List(workerConnectorList, KCTaskOverviewVO.class));
|
||||
}
|
||||
}
|
||||
@@ -1,14 +1,11 @@
|
||||
package com.xiaojukeji.know.streaming.km.biz.group;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.ClusterGroupSummaryDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.group.GroupOffsetResetDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationBaseDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationSortDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.topic.TopicPartitionKS;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.po.group.GroupMemberPO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.group.GroupOverviewVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.group.GroupTopicConsumedDetailVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.group.GroupTopicOverviewVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.exception.AdminOperateException;
|
||||
@@ -25,10 +22,6 @@ public interface GroupManager {
|
||||
String searchGroupKeyword,
|
||||
PaginationBaseDTO dto);
|
||||
|
||||
PaginationResult<GroupTopicOverviewVO> pagingGroupTopicMembers(Long clusterPhyId, String groupName, PaginationBaseDTO dto);
|
||||
|
||||
PaginationResult<GroupOverviewVO> pagingClusterGroupsOverview(Long clusterPhyId, ClusterGroupSummaryDTO dto);
|
||||
|
||||
PaginationResult<GroupTopicConsumedDetailVO> pagingGroupTopicConsumedMetrics(Long clusterPhyId,
|
||||
String topicName,
|
||||
String groupName,
|
||||
@@ -38,6 +31,4 @@ public interface GroupManager {
|
||||
Result<Set<TopicPartitionKS>> listClusterPhyGroupPartitions(Long clusterPhyId, String groupName, Long startTime, Long endTime);
|
||||
|
||||
Result<Void> resetGroupOffsets(GroupOffsetResetDTO dto, String operator) throws Exception;
|
||||
|
||||
List<GroupTopicOverviewVO> getGroupTopicOverviewVOList (Long clusterPhyId, List<GroupMemberPO> groupMemberPOList);
|
||||
}
|
||||
|
||||
@@ -3,35 +3,23 @@ package com.xiaojukeji.know.streaming.km.biz.group.impl;
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.biz.group.GroupManager;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.cluster.ClusterGroupSummaryDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.group.GroupOffsetResetDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationBaseDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationSortDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.partition.PartitionOffsetDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.group.Group;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.group.GroupTopic;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.group.GroupTopicMember;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.kafka.KSGroupDescription;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.kafka.KSMemberConsumerAssignment;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.kafka.KSMemberDescription;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.GroupMetrics;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.offset.KSOffsetSpec;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.topic.Topic;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.topic.TopicPartitionKS;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.ResultStatus;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.po.group.GroupMemberPO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.group.GroupOverviewVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.group.GroupTopicConsumedDetailVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.group.GroupTopicOverviewVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.MsgConstant;
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.PaginationConstant;
|
||||
import com.xiaojukeji.know.streaming.km.common.converter.GroupConverter;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.AggTypeEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.OffsetTypeEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.SortTypeEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.group.GroupStateEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.exception.AdminOperateException;
|
||||
import com.xiaojukeji.know.streaming.km.common.exception.NotExistException;
|
||||
@@ -39,13 +27,15 @@ import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.PaginationMetricsUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.PaginationUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterPhyService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.group.GroupMetricService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.group.GroupService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.partition.PartitionService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.topic.TopicService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.GroupMetricVersionItems;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.metrics.GroupMetricVersionItems;
|
||||
import com.xiaojukeji.know.streaming.km.persistence.es.dao.GroupMetricESDAO;
|
||||
import org.apache.kafka.clients.admin.ConsumerGroupDescription;
|
||||
import org.apache.kafka.clients.admin.MemberDescription;
|
||||
import org.apache.kafka.clients.admin.OffsetSpec;
|
||||
import org.apache.kafka.common.ConsumerGroupState;
|
||||
import org.apache.kafka.common.TopicPartition;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
@@ -54,8 +44,6 @@ import org.springframework.stereotype.Component;
|
||||
import java.util.*;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.common.enums.group.GroupTypeEnum.CONNECT_CLUSTER_PROTOCOL_TYPE;
|
||||
|
||||
@Component
|
||||
public class GroupManagerImpl implements GroupManager {
|
||||
private static final ILog log = LogFactory.getLog(GroupManagerImpl.class);
|
||||
@@ -75,9 +63,6 @@ public class GroupManagerImpl implements GroupManager {
|
||||
@Autowired
|
||||
private GroupMetricESDAO groupMetricESDAO;
|
||||
|
||||
@Autowired
|
||||
private ClusterPhyService clusterPhyService;
|
||||
|
||||
@Override
|
||||
public PaginationResult<GroupTopicOverviewVO> pagingGroupMembers(Long clusterPhyId,
|
||||
String topicName,
|
||||
@@ -86,60 +71,30 @@ public class GroupManagerImpl implements GroupManager {
|
||||
String searchGroupKeyword,
|
||||
PaginationBaseDTO dto) {
|
||||
PaginationResult<GroupMemberPO> paginationResult = groupService.pagingGroupMembers(clusterPhyId, topicName, groupName, searchTopicKeyword, searchGroupKeyword, dto);
|
||||
if (paginationResult.failed()) {
|
||||
return PaginationResult.buildFailure(paginationResult, dto);
|
||||
}
|
||||
|
||||
if (!paginationResult.hasData()) {
|
||||
return PaginationResult.buildSuc(new ArrayList<>(), paginationResult);
|
||||
}
|
||||
|
||||
List<GroupTopicOverviewVO> groupTopicVOList = this.getGroupTopicOverviewVOList(clusterPhyId, paginationResult.getData().getBizData());
|
||||
|
||||
return PaginationResult.buildSuc(groupTopicVOList, paginationResult);
|
||||
}
|
||||
|
||||
@Override
|
||||
public PaginationResult<GroupTopicOverviewVO> pagingGroupTopicMembers(Long clusterPhyId, String groupName, PaginationBaseDTO dto) {
|
||||
Group group = groupService.getGroupFromDB(clusterPhyId, groupName);
|
||||
|
||||
//没有topicMember则直接返回
|
||||
if (group == null || ValidateUtils.isEmptyList(group.getTopicMembers())) {
|
||||
return PaginationResult.buildSuc(dto);
|
||||
// 获取指标
|
||||
Result<List<GroupMetrics>> metricsListResult = groupMetricService.listLatestMetricsAggByGroupTopicFromES(
|
||||
clusterPhyId,
|
||||
paginationResult.getData().getBizData().stream().map(elem -> new GroupTopic(elem.getGroupName(), elem.getTopicName())).collect(Collectors.toList()),
|
||||
Arrays.asList(GroupMetricVersionItems.GROUP_METRIC_LAG),
|
||||
AggTypeEnum.MAX
|
||||
);
|
||||
if (metricsListResult.failed()) {
|
||||
// 如果查询失败，则输出错误信息，但是依旧进行已有数据的返回
|
||||
log.error("method=pagingGroupMembers||clusterPhyId={}||topicName={}||groupName={}||result={}||errMsg=search es failed", clusterPhyId, topicName, groupName, metricsListResult);
|
||||
}
|
||||
|
||||
//排序
|
||||
List<GroupTopicMember> groupTopicMembers = PaginationUtil.pageBySort(group.getTopicMembers(), PaginationConstant.DEFAULT_GROUP_TOPIC_SORTED_FIELD, SortTypeEnum.DESC.getSortType());
|
||||
|
||||
//分页
|
||||
PaginationResult<GroupTopicMember> paginationResult = PaginationUtil.pageBySubData(groupTopicMembers, dto);
|
||||
|
||||
List<GroupMemberPO> groupMemberPOList = paginationResult.getData().getBizData().stream().map(elem -> new GroupMemberPO(clusterPhyId, elem.getTopicName(), groupName, group.getState().getState(), elem.getMemberCount())).collect(Collectors.toList());
|
||||
|
||||
return PaginationResult.buildSuc(this.getGroupTopicOverviewVOList(clusterPhyId, groupMemberPOList), paginationResult);
|
||||
}
|
||||
|
||||
@Override
|
||||
public PaginationResult<GroupOverviewVO> pagingClusterGroupsOverview(Long clusterPhyId, ClusterGroupSummaryDTO dto) {
|
||||
List<Group> groupList = groupService.listClusterGroups(clusterPhyId);
|
||||
|
||||
// 类型转化
|
||||
List<GroupOverviewVO> voList = groupList.stream().map(elem -> GroupConverter.convert2GroupOverviewVO(elem)).collect(Collectors.toList());
|
||||
|
||||
// 搜索groupName
|
||||
voList = PaginationUtil.pageByFuzzyFilter(voList, dto.getSearchGroupName(), Arrays.asList("name"));
|
||||
|
||||
//搜索topic
|
||||
if (!ValidateUtils.isBlank(dto.getSearchTopicName())) {
|
||||
voList = voList.stream().filter(elem -> {
|
||||
for (String topicName : elem.getTopicNameList()) {
|
||||
if (topicName.contains(dto.getSearchTopicName())) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}).collect(Collectors.toList());
|
||||
}
|
||||
|
||||
// 分页 后 返回
|
||||
return PaginationUtil.pageBySubData(voList, dto);
|
||||
return PaginationResult.buildSuc(
|
||||
this.convert2GroupTopicOverviewVOList(paginationResult.getData().getBizData(), metricsListResult.getData()),
|
||||
paginationResult
|
||||
);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -148,13 +103,8 @@ public class GroupManagerImpl implements GroupManager {
|
||||
String groupName,
|
||||
List<String> latestMetricNames,
|
||||
PaginationSortDTO dto) throws NotExistException, AdminOperateException {
|
||||
ClusterPhy clusterPhy = clusterPhyService.getClusterByCluster(clusterPhyId);
|
||||
if (clusterPhy == null) {
|
||||
return PaginationResult.buildFailure(MsgConstant.getClusterPhyNotExist(clusterPhyId), dto);
|
||||
}
|
||||
|
||||
// 获取消费组消费的TopicPartition列表
|
||||
Map<TopicPartition, Long> consumedOffsetMap = groupService.getGroupOffsetFromKafka(clusterPhyId, groupName);
|
||||
Map<TopicPartition, Long> consumedOffsetMap = groupService.getGroupOffset(clusterPhyId, groupName);
|
||||
List<Integer> partitionList = consumedOffsetMap.keySet()
|
||||
.stream()
|
||||
.filter(elem -> elem.topic().equals(topicName))
|
||||
@@ -163,18 +113,13 @@ public class GroupManagerImpl implements GroupManager {
|
||||
Collections.sort(partitionList);
|
||||
|
||||
// 获取消费组当前运行信息
|
||||
KSGroupDescription groupDescription = groupService.getGroupDescriptionFromKafka(clusterPhy, groupName);
|
||||
ConsumerGroupDescription groupDescription = groupService.getGroupDescription(clusterPhyId, groupName);
|
||||
|
||||
// 转换存储格式
|
||||
Map<TopicPartition, KSMemberDescription> tpMemberMap = new HashMap<>();
|
||||
|
||||
//如果不是connect集群
|
||||
if (!groupDescription.protocolType().equals(CONNECT_CLUSTER_PROTOCOL_TYPE)) {
|
||||
for (KSMemberDescription description : groupDescription.members()) {
|
||||
KSMemberConsumerAssignment assignment = (KSMemberConsumerAssignment) description.assignment();
|
||||
for (TopicPartition tp : assignment.topicPartitions()) {
|
||||
tpMemberMap.put(tp, description);
|
||||
}
|
||||
Map<TopicPartition, MemberDescription> tpMemberMap = new HashMap<>();
|
||||
for (MemberDescription description: groupDescription.members()) {
|
||||
for (TopicPartition tp: description.assignment().topicPartitions()) {
|
||||
tpMemberMap.put(tp, description);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -191,11 +136,11 @@ public class GroupManagerImpl implements GroupManager {
|
||||
vo.setTopicName(topicName);
|
||||
vo.setPartitionId(groupMetrics.getPartitionId());
|
||||
|
||||
KSMemberDescription ksMemberDescription = tpMemberMap.get(new TopicPartition(topicName, groupMetrics.getPartitionId()));
|
||||
if (ksMemberDescription != null) {
|
||||
vo.setMemberId(ksMemberDescription.consumerId());
|
||||
vo.setHost(ksMemberDescription.host());
|
||||
vo.setClientId(ksMemberDescription.clientId());
|
||||
MemberDescription memberDescription = tpMemberMap.get(new TopicPartition(topicName, groupMetrics.getPartitionId()));
|
||||
if (memberDescription != null) {
|
||||
vo.setMemberId(memberDescription.consumerId());
|
||||
vo.setHost(memberDescription.host());
|
||||
vo.setClientId(memberDescription.clientId());
|
||||
}
|
||||
|
||||
vo.setLatestMetrics(groupMetrics);
|
||||
@@ -221,18 +166,13 @@ public class GroupManagerImpl implements GroupManager {
|
||||
return rv;
|
||||
}
|
||||
|
||||
ClusterPhy clusterPhy = clusterPhyService.getClusterByCluster(dto.getClusterId());
|
||||
if (clusterPhy == null) {
|
||||
return Result.buildFromRSAndMsg(ResultStatus.CLUSTER_NOT_EXIST, MsgConstant.getClusterPhyNotExist(dto.getClusterId()));
|
||||
}
|
||||
|
||||
KSGroupDescription description = groupService.getGroupDescriptionFromKafka(clusterPhy, dto.getGroupName());
|
||||
ConsumerGroupDescription description = groupService.getGroupDescription(dto.getClusterId(), dto.getGroupName());
|
||||
if (ConsumerGroupState.DEAD.equals(description.state()) && !dto.isCreateIfNotExist()) {
|
||||
return Result.buildFromRSAndMsg(ResultStatus.KAFKA_OPERATE_FAILED, "group不存在, 重置失败");
|
||||
}
|
||||
|
||||
if (!ConsumerGroupState.EMPTY.equals(description.state()) && !ConsumerGroupState.DEAD.equals(description.state())) {
|
||||
return Result.buildFromRSAndMsg(ResultStatus.KAFKA_OPERATE_FAILED, String.format("group处于%s, 重置失败(仅Empty | Dead 情况可重置)", GroupStateEnum.getByRawState(description.state()).getState()));
|
||||
return Result.buildFromRSAndMsg(ResultStatus.KAFKA_OPERATE_FAILED, String.format("group处于%s, 重置失败(仅Empty情况可重置)", GroupStateEnum.getByRawState(description.state()).getState()));
|
||||
}
|
||||
|
||||
// 获取offset
|
||||
@@ -245,22 +185,6 @@ public class GroupManagerImpl implements GroupManager {
|
||||
return groupService.resetGroupOffsets(dto.getClusterId(), dto.getGroupName(), offsetMapResult.getData(), operator);
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<GroupTopicOverviewVO> getGroupTopicOverviewVOList(Long clusterPhyId, List<GroupMemberPO> groupMemberPOList) {
|
||||
// 获取指标
|
||||
Result<List<GroupMetrics>> metricsListResult = groupMetricService.listLatestMetricsAggByGroupTopicFromES(
|
||||
clusterPhyId,
|
||||
groupMemberPOList.stream().map(elem -> new GroupTopic(elem.getGroupName(), elem.getTopicName())).collect(Collectors.toList()),
|
||||
Arrays.asList(GroupMetricVersionItems.GROUP_METRIC_LAG),
|
||||
AggTypeEnum.MAX
|
||||
);
|
||||
if (metricsListResult.failed()) {
|
||||
// 如果查询失败，则输出错误信息，但是依旧进行已有数据的返回
|
||||
log.error("method=completeMetricData||clusterPhyId={}||result={}||errMsg=search es failed", clusterPhyId, metricsListResult);
|
||||
}
|
||||
return this.convert2GroupTopicOverviewVOList(groupMemberPOList, metricsListResult.getData());
|
||||
}
|
||||
|
||||
|
||||
/**************************************************** private method ****************************************************/
|
||||
|
||||
@@ -297,16 +221,16 @@ public class GroupManagerImpl implements GroupManager {
|
||||
)));
|
||||
}
|
||||
|
||||
KSOffsetSpec offsetSpec = null;
|
||||
OffsetSpec offsetSpec = null;
|
||||
if (OffsetTypeEnum.PRECISE_TIMESTAMP.getResetType() == dto.getResetType()) {
|
||||
offsetSpec = KSOffsetSpec.forTimestamp(dto.getTimestamp());
|
||||
offsetSpec = OffsetSpec.forTimestamp(dto.getTimestamp());
|
||||
} else if (OffsetTypeEnum.EARLIEST.getResetType() == dto.getResetType()) {
|
||||
offsetSpec = KSOffsetSpec.earliest();
|
||||
offsetSpec = OffsetSpec.earliest();
|
||||
} else {
|
||||
offsetSpec = KSOffsetSpec.latest();
|
||||
offsetSpec = OffsetSpec.latest();
|
||||
}
|
||||
|
||||
return partitionService.getPartitionOffsetFromKafka(dto.getClusterId(), dto.getTopicName(), offsetSpec);
|
||||
return partitionService.getPartitionOffsetFromKafka(dto.getClusterId(), dto.getTopicName(), offsetSpec, dto.getTimestamp());
|
||||
}
|
||||
|
||||
private List<GroupTopicOverviewVO> convert2GroupTopicOverviewVOList(List<GroupMemberPO> poList, List<GroupMetrics> metricsList) {
|
||||
@@ -368,4 +292,5 @@ public class GroupManagerImpl implements GroupManager {
|
||||
dto
|
||||
);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -22,7 +22,7 @@ import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.reassign.ReassignService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.topic.TopicMetricService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.topic.TopicService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.TopicMetricVersionItems;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.metrics.TopicMetricVersionItems;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
|
||||
@@ -1,10 +1,8 @@
|
||||
package com.xiaojukeji.know.streaming.km.biz.topic;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationBaseDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationSortDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.topic.TopicRecordDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.group.GroupTopicOverviewVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.topic.TopicBrokersPartitionsSummaryVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.topic.TopicRecordVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.topic.TopicStateVO;
|
||||
@@ -25,6 +23,4 @@ public interface TopicStateManager {
|
||||
Result<List<TopicPartitionVO>> getTopicPartitions(Long clusterPhyId, String topicName, List<String> metricsNames);
|
||||
|
||||
Result<TopicBrokersPartitionsSummaryVO> getTopicBrokersPartitionsSummary(Long clusterPhyId, String topicName);
|
||||
|
||||
PaginationResult<GroupTopicOverviewVO> pagingTopicGroupsOverview(Long clusterPhyId, String topicName, String searchGroupName, PaginationBaseDTO dto);
|
||||
}
|
||||
|
||||
@@ -10,18 +10,14 @@ import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.param.topic.TopicCreateParam;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.param.topic.TopicParam;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.param.topic.TopicPartitionExpandParam;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.partition.Partition;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.ResultStatus;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.topic.Topic;
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.MsgConstant;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.BackoffUtils;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.FutureUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.kafka.KafkaReplicaAssignUtil;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.broker.BrokerService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterPhyService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.partition.PartitionService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.topic.OpTopicService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.topic.TopicService;
|
||||
import kafka.admin.AdminUtils;
|
||||
@@ -56,9 +52,6 @@ public class OpTopicManagerImpl implements OpTopicManager {
|
||||
@Autowired
|
||||
private ClusterPhyService clusterPhyService;
|
||||
|
||||
@Autowired
|
||||
private PartitionService partitionService;
|
||||
|
||||
@Override
|
||||
public Result<Void> createTopic(TopicCreateDTO dto, String operator) {
|
||||
log.info("method=createTopic||param={}||operator={}.", dto, operator);
|
||||
@@ -87,7 +80,7 @@ public class OpTopicManagerImpl implements OpTopicManager {
|
||||
);
|
||||
|
||||
// 创建Topic
|
||||
Result<Void> createTopicRes = opTopicService.createTopic(
|
||||
return opTopicService.createTopic(
|
||||
new TopicCreateParam(
|
||||
dto.getClusterId(),
|
||||
dto.getTopicName(),
|
||||
@@ -97,21 +90,6 @@ public class OpTopicManagerImpl implements OpTopicManager {
|
||||
),
|
||||
operator
|
||||
);
|
||||
if (createTopicRes.successful()){
|
||||
try{
|
||||
FutureUtil.quickStartupFutureUtil.submitTask(() -> {
|
||||
BackoffUtils.backoff(3000);
|
||||
Result<List<Partition>> partitionsResult = partitionService.listPartitionsFromKafka(clusterPhy, dto.getTopicName());
|
||||
if (partitionsResult.successful()){
|
||||
partitionService.updatePartitions(clusterPhy.getId(), dto.getTopicName(), partitionsResult.getData(), new ArrayList<>());
|
||||
}
|
||||
});
|
||||
}catch (Exception e) {
|
||||
log.error("method=createTopic||param={}||operator={}||msg=add partition to db failed||errMsg=exception", dto, operator, e);
|
||||
return Result.buildFromRSAndMsg(ResultStatus.MYSQL_OPERATE_FAILED, "Topic创建成功，但记录Partition到DB中失败，等待定时任务同步partition信息");
|
||||
}
|
||||
}
|
||||
return createTopicRes;
|
||||
}
|
||||
|
||||
@Override
|
||||
|
||||
@@ -16,7 +16,7 @@ import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.broker.BrokerConfigService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.broker.BrokerService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.topic.TopicConfigService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.BaseKafkaVersionControlService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.BaseVersionControlService;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
@@ -27,7 +27,7 @@ import java.util.stream.Collectors;
|
||||
import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionEnum.*;
|
||||
|
||||
@Component
|
||||
public class TopicConfigManagerImpl extends BaseKafkaVersionControlService implements TopicConfigManager {
|
||||
public class TopicConfigManagerImpl extends BaseVersionControlService implements TopicConfigManager {
|
||||
private static final ILog log = LogFactory.getLog(TopicConfigManagerImpl.class);
|
||||
|
||||
private static final String GET_DEFAULT_TOPIC_CONFIG = "getDefaultTopicConfig";
|
||||
|
||||
@@ -2,23 +2,17 @@ package com.xiaojukeji.know.streaming.km.biz.topic.impl;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.biz.group.GroupManager;
|
||||
import com.xiaojukeji.know.streaming.km.biz.topic.TopicStateManager;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationBaseDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.topic.TopicRecordDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.broker.Broker;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.PartitionMetrics;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.TopicMetrics;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.offset.KSOffsetSpec;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.partition.Partition;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.PaginationResult;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.ResultStatus;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.topic.Topic;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.po.group.GroupMemberPO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.broker.BrokerReplicaSummaryVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.group.GroupTopicOverviewVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.topic.TopicBrokersPartitionsSummaryVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.topic.TopicRecordVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.topic.TopicStateVO;
|
||||
@@ -38,15 +32,15 @@ import com.xiaojukeji.know.streaming.km.common.utils.PaginationUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.broker.BrokerService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterPhyService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.group.GroupService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.partition.PartitionMetricService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.partition.PartitionService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.topic.TopicConfigService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.topic.TopicMetricService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.topic.TopicService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.TopicMetricVersionItems;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.metrics.TopicMetricVersionItems;
|
||||
import org.apache.commons.lang3.ObjectUtils;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.apache.kafka.clients.admin.OffsetSpec;
|
||||
import org.apache.kafka.clients.consumer.*;
|
||||
import org.apache.kafka.common.TopicPartition;
|
||||
import org.apache.kafka.common.config.TopicConfig;
|
||||
@@ -83,12 +77,6 @@ public class TopicStateManagerImpl implements TopicStateManager {
|
||||
@Autowired
|
||||
private TopicConfigService topicConfigService;
|
||||
|
||||
@Autowired
|
||||
private GroupService groupService;
|
||||
|
||||
@Autowired
|
||||
private GroupManager groupManager;
|
||||
|
||||
@Override
|
||||
public TopicBrokerAllVO getTopicBrokerAll(Long clusterPhyId, String topicName, String searchBrokerHost) throws NotExistException {
|
||||
Topic topic = topicService.getTopic(clusterPhyId, topicName);
|
||||
@@ -143,12 +131,12 @@ public class TopicStateManagerImpl implements TopicStateManager {
|
||||
}
|
||||
|
||||
// 获取分区beginOffset
|
||||
Result<Map<TopicPartition, Long>> beginOffsetsMapResult = partitionService.getPartitionOffsetFromKafka(clusterPhyId, topicName, dto.getFilterPartitionId(), KSOffsetSpec.earliest());
|
||||
Result<Map<TopicPartition, Long>> beginOffsetsMapResult = partitionService.getPartitionOffsetFromKafka(clusterPhyId, topicName, dto.getFilterPartitionId(), OffsetSpec.earliest(), null);
|
||||
if (beginOffsetsMapResult.failed()) {
|
||||
return Result.buildFromIgnoreData(beginOffsetsMapResult);
|
||||
}
|
||||
// 获取分区endOffset
|
||||
Result<Map<TopicPartition, Long>> endOffsetsMapResult = partitionService.getPartitionOffsetFromKafka(clusterPhyId, topicName, dto.getFilterPartitionId(), KSOffsetSpec.latest());
|
||||
Result<Map<TopicPartition, Long>> endOffsetsMapResult = partitionService.getPartitionOffsetFromKafka(clusterPhyId, topicName, dto.getFilterPartitionId(), OffsetSpec.latest(), null);
|
||||
if (endOffsetsMapResult.failed()) {
|
||||
return Result.buildFromIgnoreData(endOffsetsMapResult);
|
||||
}
|
||||
@@ -307,7 +295,7 @@ public class TopicStateManagerImpl implements TopicStateManager {
|
||||
if (metricsResult.failed()) {
|
||||
// 仅打印错误日志，但是不直接返回错误
|
||||
log.error(
|
||||
"method=getTopicPartitions||clusterPhyId={}||topicName={}||result={}||msg=get metrics from es failed",
|
||||
"class=TopicStateManagerImpl||method=getTopicPartitions||clusterPhyId={}||topicName={}||result={}||msg=get metrics from es failed",
|
||||
clusterPhyId, topicName, metricsResult
|
||||
);
|
||||
}
|
||||
@@ -358,19 +346,6 @@ public class TopicStateManagerImpl implements TopicStateManager {
|
||||
return Result.buildSuc(vo);
|
||||
}
|
||||
|
||||
@Override
|
||||
public PaginationResult<GroupTopicOverviewVO> pagingTopicGroupsOverview(Long clusterPhyId, String topicName, String searchGroupName, PaginationBaseDTO dto) {
|
||||
PaginationResult<GroupMemberPO> paginationResult = groupService.pagingGroupMembers(clusterPhyId, topicName, "", "", searchGroupName, dto);
|
||||
|
||||
if (!paginationResult.hasData()) {
|
||||
return PaginationResult.buildSuc(new ArrayList<>(), paginationResult);
|
||||
}
|
||||
|
||||
List<GroupTopicOverviewVO> groupTopicVOList = groupManager.getGroupTopicOverviewVOList(clusterPhyId, paginationResult.getData().getBizData());
|
||||
|
||||
return PaginationResult.buildSuc(groupTopicVOList, paginationResult);
|
||||
}
|
||||
|
||||
/**************************************************** private method ****************************************************/
|
||||
|
||||
private boolean checkIfIgnore(ConsumerRecord<String, String> consumerRecord, String filterKey, String filterValue) {
|
||||
|
||||
@@ -20,7 +20,7 @@ public interface VersionControlManager {
|
||||
* 获取当前ks所有支持的kafka版本
|
||||
* @return
|
||||
*/
|
||||
Result<Map<String, Long>> listAllKafkaVersions();
|
||||
Result<Map<String, Long>> listAllVersions();
|
||||
|
||||
/**
|
||||
* 获取全部集群 clusterId 中类型为 type 的指标，不论支持不支持
|
||||
@@ -28,7 +28,7 @@ public interface VersionControlManager {
|
||||
* @param type
|
||||
* @return
|
||||
*/
|
||||
Result<List<VersionItemVO>> listKafkaClusterVersionControlItem(Long clusterId, Integer type);
|
||||
Result<List<VersionItemVO>> listClusterVersionControlItem(Long clusterId, Integer type);
|
||||
|
||||
/**
|
||||
* 获取当前用户设置的用于展示的指标配置
|
||||
|
||||
@@ -14,10 +14,10 @@ import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionControlItem;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.config.metric.UserMetricConfigVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.vo.version.VersionItemVO;
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.Constant;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.version.VersionEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.VersionUtil;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterPhyService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.VersionControlService;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Service;
|
||||
@@ -30,10 +30,10 @@ import java.util.stream.Collectors;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionEnum.V_MAX;
|
||||
import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum.*;
|
||||
import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.BrokerMetricVersionItems.*;
|
||||
import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.ClusterMetricVersionItems.*;
|
||||
import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.GroupMetricVersionItems.*;
|
||||
import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.kafka.TopicMetricVersionItems.*;
|
||||
import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.BrokerMetricVersionItems.*;
|
||||
import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.ClusterMetricVersionItems.*;
|
||||
import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.GroupMetricVersionItems.*;
|
||||
import static com.xiaojukeji.know.streaming.km.core.service.version.metrics.TopicMetricVersionItems.*;
|
||||
|
||||
@Service
|
||||
public class VersionControlManagerImpl implements VersionControlManager {
|
||||
@@ -48,7 +48,7 @@ public class VersionControlManagerImpl implements VersionControlManager {
|
||||
|
||||
@PostConstruct
|
||||
public void init(){
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_TOPIC.getCode(), TOPIC_METRIC_HEALTH_STATE, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_TOPIC.getCode(), TOPIC_METRIC_HEALTH_SCORE, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_TOPIC.getCode(), TOPIC_METRIC_FAILED_FETCH_REQ, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_TOPIC.getCode(), TOPIC_METRIC_FAILED_PRODUCE_REQ, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_TOPIC.getCode(), TOPIC_METRIC_UNDER_REPLICA_PARTITIONS, true));
|
||||
@@ -58,7 +58,7 @@ public class VersionControlManagerImpl implements VersionControlManager {
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_TOPIC.getCode(), TOPIC_METRIC_BYTES_REJECTED, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_TOPIC.getCode(), TOPIC_METRIC_MESSAGE_IN, true));
|
||||
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_CLUSTER.getCode(), CLUSTER_METRIC_HEALTH_STATE, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_CLUSTER.getCode(), CLUSTER_METRIC_HEALTH_SCORE, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_CLUSTER.getCode(), CLUSTER_METRIC_ACTIVE_CONTROLLER_COUNT, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_CLUSTER.getCode(), CLUSTER_METRIC_BYTES_IN, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_CLUSTER.getCode(), CLUSTER_METRIC_BYTES_OUT, true));
|
||||
@@ -76,9 +76,9 @@ public class VersionControlManagerImpl implements VersionControlManager {
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_GROUP.getCode(), GROUP_METRIC_OFFSET_CONSUMED, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_GROUP.getCode(), GROUP_METRIC_LAG, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_GROUP.getCode(), GROUP_METRIC_STATE, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_GROUP.getCode(), GROUP_METRIC_HEALTH_STATE, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_GROUP.getCode(), GROUP_METRIC_HEALTH_SCORE, true));
|
||||
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_BROKER.getCode(), BROKER_METRIC_HEALTH_STATE, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_BROKER.getCode(), BROKER_METRIC_HEALTH_SCORE, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_BROKER.getCode(), BROKER_METRIC_CONNECTION_COUNT, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_BROKER.getCode(), BROKER_METRIC_MESSAGE_IN, true));
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_BROKER.getCode(), BROKER_METRIC_NETWORK_RPO_AVG_IDLE, true));
|
||||
@@ -93,9 +93,6 @@ public class VersionControlManagerImpl implements VersionControlManager {
|
||||
defaultMetrics.add(new UserMetricConfig(METRIC_BROKER.getCode(), BROKER_METRIC_BYTES_OUT, true));
|
||||
}
|
||||
|
||||
@Autowired
|
||||
private ClusterPhyService clusterPhyService;
|
||||
|
||||
@Autowired
|
||||
private VersionControlService versionControlService;
|
||||
|
||||
@@ -111,40 +108,27 @@ public class VersionControlManagerImpl implements VersionControlManager {
|
||||
allVersionItemVO.addAll(ConvertUtil.list2List(versionControlService.listVersionControlItem(METRIC_BROKER.getCode()), VersionItemVO.class));
|
||||
allVersionItemVO.addAll(ConvertUtil.list2List(versionControlService.listVersionControlItem(METRIC_PARTITION.getCode()), VersionItemVO.class));
|
||||
allVersionItemVO.addAll(ConvertUtil.list2List(versionControlService.listVersionControlItem(METRIC_REPLICATION.getCode()), VersionItemVO.class));
|
||||
|
||||
allVersionItemVO.addAll(ConvertUtil.list2List(versionControlService.listVersionControlItem(METRIC_ZOOKEEPER.getCode()), VersionItemVO.class));
|
||||
|
||||
allVersionItemVO.addAll(ConvertUtil.list2List(versionControlService.listVersionControlItem(METRIC_CONNECT_CLUSTER.getCode()), VersionItemVO.class));
|
||||
allVersionItemVO.addAll(ConvertUtil.list2List(versionControlService.listVersionControlItem(METRIC_CONNECT_CONNECTOR.getCode()), VersionItemVO.class));
|
||||
allVersionItemVO.addAll(ConvertUtil.list2List(versionControlService.listVersionControlItem(METRIC_CONNECT_MIRROR_MAKER.getCode()), VersionItemVO.class));
|
||||
|
||||
allVersionItemVO.addAll(ConvertUtil.list2List(versionControlService.listVersionControlItem(WEB_OP.getCode()), VersionItemVO.class));
|
||||
|
||||
Map<String, VersionItemVO> map = allVersionItemVO.stream().collect(
|
||||
Collectors.toMap(
|
||||
u -> u.getType() + "@" + u.getName(),
|
||||
Function.identity(),
|
||||
(v1, v2) -> v1)
|
||||
);
|
||||
Collectors.toMap(u -> u.getType() + "@" + u.getName(), Function.identity() ));
|
||||
|
||||
return Result.buildSuc(map);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Result<Map<String, Long>> listAllKafkaVersions() {
|
||||
public Result<Map<String, Long>> listAllVersions() {
|
||||
return Result.buildSuc(VersionEnum.allVersionsWithOutMax());
|
||||
}
|
||||
|
||||
@Override
|
||||
public Result<List<VersionItemVO>> listKafkaClusterVersionControlItem(Long clusterId, Integer type) {
|
||||
public Result<List<VersionItemVO>> listClusterVersionControlItem(Long clusterId, Integer type) {
|
||||
List<VersionControlItem> allItem = versionControlService.listVersionControlItem(type);
|
||||
List<VersionItemVO> versionItemVOS = new ArrayList<>();
|
||||
|
||||
String versionStr = clusterPhyService.getVersionFromCacheFirst(clusterId);
|
||||
|
||||
for (VersionControlItem item : allItem){
|
||||
VersionItemVO itemVO = ConvertUtil.obj2Obj(item, VersionItemVO.class);
|
||||
boolean support = versionControlService.isClusterSupport(versionStr, item);
|
||||
boolean support = versionControlService.isClusterSupport(clusterId, item);
|
||||
|
||||
itemVO.setSupport(support);
|
||||
itemVO.setDesc(itemSupportDesc(item, support));
|
||||
@@ -157,7 +141,7 @@ public class VersionControlManagerImpl implements VersionControlManager {
|
||||
|
||||
@Override
|
||||
public Result<List<UserMetricConfigVO>> listUserMetricItem(Long clusterId, Integer type, String operator) {
|
||||
Result<List<VersionItemVO>> ret = listKafkaClusterVersionControlItem(clusterId, type);
|
||||
Result<List<VersionItemVO>> ret = listClusterVersionControlItem(clusterId, type);
|
||||
if(null == ret || ret.failed()){
|
||||
return Result.buildFail();
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.collector.service.CollectThreadPoolService;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.BaseMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.component.SpringTool;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
|
||||
@@ -8,20 +9,17 @@ import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
|
||||
|
||||
|
||||
/**
|
||||
* @author didi
|
||||
*/
|
||||
public abstract class AbstractMetricCollector<M, C> {
|
||||
public abstract String getClusterVersion(C c);
|
||||
public abstract class AbstractMetricCollector<T> {
|
||||
public abstract void collectMetrics(ClusterPhy clusterPhy);
|
||||
|
||||
public abstract VersionItemTypeEnum collectorType();
|
||||
|
||||
@Autowired
|
||||
private CollectThreadPoolService collectThreadPoolService;
|
||||
|
||||
public abstract void collectMetrics(C c);
|
||||
|
||||
protected FutureWaitUtil<Void> getFutureUtilByClusterPhyId(Long clusterPhyId) {
|
||||
return collectThreadPoolService.selectSuitableFutureUtil(clusterPhyId * 1000L + this.collectorType().getCode());
|
||||
}
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric.kafka;
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric;
|
||||
|
||||
import com.alibaba.fastjson.JSON;
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.broker.Broker;
|
||||
@@ -10,6 +11,7 @@ import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionContro
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.BrokerMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.Constant;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.EnvUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.broker.BrokerMetricService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.broker.BrokerService;
|
||||
@@ -26,8 +28,8 @@ import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemT
|
||||
* @author didi
|
||||
*/
|
||||
@Component
|
||||
public class BrokerMetricCollector extends AbstractKafkaMetricCollector<BrokerMetrics> {
|
||||
private static final ILog LOGGER = LogFactory.getLog(BrokerMetricCollector.class);
|
||||
public class BrokerMetricCollector extends AbstractMetricCollector<BrokerMetrics> {
|
||||
protected static final ILog LOGGER = LogFactory.getLog("METRIC_LOGGER");
|
||||
|
||||
@Autowired
|
||||
private VersionControlService versionControlService;
|
||||
@@ -39,31 +41,32 @@ public class BrokerMetricCollector extends AbstractKafkaMetricCollector<BrokerMe
|
||||
private BrokerService brokerService;
|
||||
|
||||
@Override
|
||||
public List<BrokerMetrics> collectKafkaMetrics(ClusterPhy clusterPhy) {
|
||||
public void collectMetrics(ClusterPhy clusterPhy) {
|
||||
Long startTime = System.currentTimeMillis();
|
||||
Long clusterPhyId = clusterPhy.getId();
|
||||
|
||||
List<Broker> brokers = brokerService.listAliveBrokersFromDB(clusterPhy.getId());
|
||||
List<VersionControlItem> items = versionControlService.listVersionControlItem(this.getClusterVersion(clusterPhy), collectorType().getCode());
|
||||
List<VersionControlItem> items = versionControlService.listVersionControlItem(clusterPhyId, collectorType().getCode());
|
||||
|
||||
FutureWaitUtil<Void> future = this.getFutureUtilByClusterPhyId(clusterPhyId);
|
||||
|
||||
List<BrokerMetrics> metricsList = new ArrayList<>();
|
||||
List<BrokerMetrics> brokerMetrics = new ArrayList<>();
|
||||
for(Broker broker : brokers) {
|
||||
BrokerMetrics metrics = new BrokerMetrics(clusterPhyId, broker.getBrokerId(), broker.getHost(), broker.getPort());
|
||||
metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
|
||||
metricsList.add(metrics);
|
||||
brokerMetrics.add(metrics);
|
||||
|
||||
future.runnableTask(
|
||||
String.format("class=BrokerMetricCollector||clusterPhyId=%d||brokerId=%d", clusterPhyId, broker.getBrokerId()),
|
||||
String.format("method=BrokerMetricCollector||clusterPhyId=%d||brokerId=%d", clusterPhyId, broker.getBrokerId()),
|
||||
30000,
|
||||
() -> collectMetrics(clusterPhyId, metrics, items)
|
||||
);
|
||||
}
|
||||
|
||||
future.waitExecute(30000);
|
||||
this.publishMetric(new BrokerMetricEvent(this, metricsList));
|
||||
this.publishMetric(new BrokerMetricEvent(this, brokerMetrics));
|
||||
|
||||
return metricsList;
|
||||
LOGGER.info("method=BrokerMetricCollector||clusterPhyId={}||startTime={}||costTime={}||msg=collect finished.",
|
||||
clusterPhyId, startTime, System.currentTimeMillis() - startTime);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -75,6 +78,7 @@ public class BrokerMetricCollector extends AbstractKafkaMetricCollector<BrokerMe
|
||||
|
||||
private void collectMetrics(Long clusterPhyId, BrokerMetrics metrics, List<VersionControlItem> items) {
|
||||
long startTime = System.currentTimeMillis();
|
||||
metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
|
||||
|
||||
for(VersionControlItem v : items) {
|
||||
try {
|
||||
@@ -88,11 +92,14 @@ public class BrokerMetricCollector extends AbstractKafkaMetricCollector<BrokerMe
|
||||
}
|
||||
|
||||
metrics.putMetric(ret.getData().getMetrics());
|
||||
|
||||
if(!EnvUtil.isOnline()){
|
||||
LOGGER.info("method=BrokerMetricCollector||clusterId={}||brokerId={}||metric={}||metric={}!",
|
||||
clusterPhyId, metrics.getBrokerId(), v.getName(), JSON.toJSONString(ret.getData().getMetrics()));
|
||||
}
|
||||
} catch (Exception e){
|
||||
LOGGER.error(
|
||||
"method=collectMetrics||clusterPhyId={}||brokerId={}||metricName={}||errMsg=exception!",
|
||||
clusterPhyId, metrics.getBrokerId(), v.getName(), e
|
||||
);
|
||||
LOGGER.error("method=BrokerMetricCollector||clusterId={}||brokerId={}||metric={}||errMsg=exception!",
|
||||
clusterPhyId, metrics.getBrokerId(), v.getName(), e);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric.kafka;
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
@@ -7,15 +7,18 @@ import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.ClusterMetric
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionControlItem;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.ClusterMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.ClusterMetricPO;
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.Constant;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.EnvUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterMetricService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.VersionControlService;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.util.Collections;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum.METRIC_CLUSTER;
|
||||
@@ -24,8 +27,8 @@ import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemT
|
||||
* @author didi
|
||||
*/
|
||||
@Component
|
||||
public class ClusterMetricCollector extends AbstractKafkaMetricCollector<ClusterMetrics> {
|
||||
protected static final ILog LOGGER = LogFactory.getLog(ClusterMetricCollector.class);
|
||||
public class ClusterMetricCollector extends AbstractMetricCollector<ClusterMetricPO> {
|
||||
protected static final ILog LOGGER = LogFactory.getLog("METRIC_LOGGER");
|
||||
|
||||
@Autowired
|
||||
private VersionControlService versionControlService;
|
||||
@@ -34,37 +37,35 @@ public class ClusterMetricCollector extends AbstractKafkaMetricCollector<Cluster
|
||||
private ClusterMetricService clusterMetricService;
|
||||
|
||||
@Override
|
||||
public List<ClusterMetrics> collectKafkaMetrics(ClusterPhy clusterPhy) {
|
||||
public void collectMetrics(ClusterPhy clusterPhy) {
|
||||
Long startTime = System.currentTimeMillis();
|
||||
Long clusterPhyId = clusterPhy.getId();
|
||||
List<VersionControlItem> items = versionControlService.listVersionControlItem(this.getClusterVersion(clusterPhy), collectorType().getCode());
|
||||
List<VersionControlItem> items = versionControlService.listVersionControlItem(clusterPhyId, collectorType().getCode());
|
||||
|
||||
ClusterMetrics metrics = new ClusterMetrics(clusterPhyId, clusterPhy.getKafkaVersion());
|
||||
metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
|
||||
|
||||
FutureWaitUtil<Void> future = this.getFutureUtilByClusterPhyId(clusterPhyId);
|
||||
|
||||
for(VersionControlItem v : items) {
|
||||
future.runnableTask(
|
||||
String.format("class=ClusterMetricCollector||clusterPhyId=%d||metricName=%s", clusterPhyId, v.getName()),
|
||||
String.format("method=ClusterMetricCollector||clusterPhyId=%d||metricName=%s", clusterPhyId, v.getName()),
|
||||
30000,
|
||||
() -> {
|
||||
try {
|
||||
if(null != metrics.getMetrics().get(v.getName())){
|
||||
return null;
|
||||
}
|
||||
if(null != metrics.getMetrics().get(v.getName())){return null;}
|
||||
|
||||
Result<ClusterMetrics> ret = clusterMetricService.collectClusterMetricsFromKafka(clusterPhyId, v.getName());
|
||||
if(null == ret || ret.failed() || null == ret.getData()){
|
||||
return null;
|
||||
}
|
||||
if(null == ret || ret.failed() || null == ret.getData()){return null;}
|
||||
|
||||
metrics.putMetric(ret.getData().getMetrics());
|
||||
|
||||
if(!EnvUtil.isOnline()){
|
||||
LOGGER.info("method=ClusterMetricCollector||clusterPhyId={}||metricName={}||metricValue={}",
|
||||
clusterPhyId, v.getName(), ConvertUtil.obj2Json(ret.getData().getMetrics()));
|
||||
}
|
||||
} catch (Exception e){
|
||||
LOGGER.error(
|
||||
"method=collectKafkaMetrics||clusterPhyId={}||metricName={}||errMsg=exception!",
|
||||
clusterPhyId, v.getName(), e
|
||||
);
|
||||
LOGGER.error("method=ClusterMetricCollector||clusterPhyId={}||metricName={}||errMsg=exception!",
|
||||
clusterPhyId, v.getName(), e);
|
||||
}
|
||||
|
||||
return null;
|
||||
@@ -75,9 +76,10 @@ public class ClusterMetricCollector extends AbstractKafkaMetricCollector<Cluster
|
||||
|
||||
metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, (System.currentTimeMillis() - startTime) / 1000.0f);
|
||||
|
||||
publishMetric(new ClusterMetricEvent(this, Collections.singletonList(metrics)));
|
||||
publishMetric(new ClusterMetricEvent(this, Arrays.asList(metrics)));
|
||||
|
||||
return Collections.singletonList(metrics);
|
||||
LOGGER.info("method=ClusterMetricCollector||clusterPhyId={}||startTime={}||costTime={}||msg=msg=collect finished.",
|
||||
clusterPhyId, startTime, System.currentTimeMillis() - startTime);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -1,5 +1,6 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric.kafka;
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric;
|
||||
|
||||
import com.alibaba.fastjson.JSON;
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
|
||||
@@ -9,16 +10,20 @@ import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionContro
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.GroupMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.Constant;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.EnvUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.group.GroupMetricService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.group.GroupService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.VersionControlService;
|
||||
import org.apache.kafka.common.TopicPartition;
|
||||
import org.apache.commons.collections.CollectionUtils;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum.METRIC_GROUP;
|
||||
@@ -27,8 +32,8 @@ import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemT
|
||||
* @author didi
|
||||
*/
|
||||
@Component
|
||||
public class GroupMetricCollector extends AbstractKafkaMetricCollector<GroupMetrics> {
|
||||
protected static final ILog LOGGER = LogFactory.getLog(GroupMetricCollector.class);
|
||||
public class GroupMetricCollector extends AbstractMetricCollector<List<GroupMetrics>> {
|
||||
protected static final ILog LOGGER = LogFactory.getLog("METRIC_LOGGER");
|
||||
|
||||
@Autowired
|
||||
private VersionControlService versionControlService;
|
||||
@@ -40,38 +45,40 @@ public class GroupMetricCollector extends AbstractKafkaMetricCollector<GroupMetr
|
||||
private GroupService groupService;
|
||||
|
||||
@Override
|
||||
public List<GroupMetrics> collectKafkaMetrics(ClusterPhy clusterPhy) {
|
||||
public void collectMetrics(ClusterPhy clusterPhy) {
|
||||
Long startTime = System.currentTimeMillis();
|
||||
Long clusterPhyId = clusterPhy.getId();
|
||||
|
||||
List<String> groupNameList = new ArrayList<>();
|
||||
List<String> groups = new ArrayList<>();
|
||||
try {
|
||||
groupNameList = groupService.listGroupsFromKafka(clusterPhy);
|
||||
groups = groupService.listGroupsFromKafka(clusterPhyId);
|
||||
} catch (Exception e) {
|
||||
LOGGER.error("method=collectKafkaMetrics||clusterPhyId={}||msg=exception!", clusterPhyId, e);
|
||||
LOGGER.error("method=GroupMetricCollector||clusterPhyId={}||msg=exception!", clusterPhyId, e);
|
||||
}
|
||||
|
||||
if(ValidateUtils.isEmptyList(groupNameList)) {
|
||||
return Collections.emptyList();
|
||||
}
|
||||
if(CollectionUtils.isEmpty(groups)){return;}
|
||||
|
||||
List<VersionControlItem> items = versionControlService.listVersionControlItem(this.getClusterVersion(clusterPhy), collectorType().getCode());
|
||||
List<VersionControlItem> items = versionControlService.listVersionControlItem(clusterPhyId, collectorType().getCode());
|
||||
|
||||
FutureWaitUtil<Void> future = this.getFutureUtilByClusterPhyId(clusterPhyId);
|
||||
FutureWaitUtil<Void> future = getFutureUtilByClusterPhyId(clusterPhyId);
|
||||
|
||||
Map<String, List<GroupMetrics>> metricsMap = new ConcurrentHashMap<>();
|
||||
for(String groupName : groupNameList) {
|
||||
for(String groupName : groups) {
|
||||
future.runnableTask(
|
||||
String.format("class=GroupMetricCollector||clusterPhyId=%d||groupName=%s", clusterPhyId, groupName),
|
||||
String.format("method=GroupMetricCollector||clusterPhyId=%d||groupName=%s", clusterPhyId, groupName),
|
||||
30000,
|
||||
() -> collectMetrics(clusterPhyId, groupName, metricsMap, items));
|
||||
}
|
||||
|
||||
future.waitResult(30000);
|
||||
|
||||
List<GroupMetrics> metricsList = metricsMap.values().stream().collect(ArrayList::new, ArrayList::addAll, ArrayList::addAll);
|
||||
List<GroupMetrics> metricsList = new ArrayList<>();
|
||||
metricsMap.values().forEach(elem -> metricsList.addAll(elem));
|
||||
|
||||
publishMetric(new GroupMetricEvent(this, metricsList));
|
||||
return metricsList;
|
||||
|
||||
LOGGER.info("method=GroupMetricCollector||clusterPhyId={}||startTime={}||cost={}||msg=collect finished.",
|
||||
clusterPhyId, startTime, System.currentTimeMillis() - startTime);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -84,7 +91,9 @@ public class GroupMetricCollector extends AbstractKafkaMetricCollector<GroupMetr
|
||||
private void collectMetrics(Long clusterPhyId, String groupName, Map<String, List<GroupMetrics>> metricsMap, List<VersionControlItem> items) {
|
||||
long startTime = System.currentTimeMillis();
|
||||
|
||||
Map<TopicPartition, GroupMetrics> subMetricMap = new HashMap<>();
|
||||
List<GroupMetrics> groupMetricsList = new ArrayList<>();
|
||||
|
||||
Map<String, GroupMetrics> tpGroupPOMap = new HashMap<>();
|
||||
|
||||
GroupMetrics groupMetrics = new GroupMetrics(clusterPhyId, groupName, true);
|
||||
groupMetrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
|
||||
@@ -98,31 +107,38 @@ public class GroupMetricCollector extends AbstractKafkaMetricCollector<GroupMetr
|
||||
continue;
|
||||
}
|
||||
|
||||
ret.getData().forEach(metrics -> {
|
||||
ret.getData().stream().forEach(metrics -> {
|
||||
if (metrics.isBGroupMetric()) {
|
||||
groupMetrics.putMetric(metrics.getMetrics());
|
||||
return;
|
||||
}
|
||||
} else {
|
||||
String topicName = metrics.getTopic();
|
||||
Integer partitionId = metrics.getPartitionId();
|
||||
String tpGroupKey = genTopicPartitionGroupKey(topicName, partitionId);
|
||||
|
||||
TopicPartition tp = new TopicPartition(metrics.getTopic(), metrics.getPartitionId());
|
||||
subMetricMap.putIfAbsent(tp, new GroupMetrics(clusterPhyId, metrics.getPartitionId(), metrics.getTopic(), groupName, false));
|
||||
subMetricMap.get(tp).putMetric(metrics.getMetrics());
|
||||
tpGroupPOMap.putIfAbsent(tpGroupKey, new GroupMetrics(clusterPhyId, partitionId, topicName, groupName, false));
|
||||
tpGroupPOMap.get(tpGroupKey).putMetric(metrics.getMetrics());
|
||||
}
|
||||
});
|
||||
} catch (Exception e) {
|
||||
LOGGER.error(
|
||||
"method=collectMetrics||clusterPhyId={}||groupName={}||errMsg=exception!",
|
||||
clusterPhyId, groupName, e
|
||||
);
|
||||
|
||||
if(!EnvUtil.isOnline()){
|
||||
LOGGER.info("method=GroupMetricCollector||clusterPhyId={}||groupName={}||metricName={}||metricValue={}",
|
||||
clusterPhyId, groupName, metricName, JSON.toJSONString(ret.getData()));
|
||||
}
|
||||
}catch (Exception e){
|
||||
LOGGER.error("method=GroupMetricCollector||clusterPhyId={}||groupName={}||errMsg=exception!", clusterPhyId, groupName, e);
|
||||
}
|
||||
}
|
||||
|
||||
List<GroupMetrics> metricsList = new ArrayList<>();
|
||||
metricsList.add(groupMetrics);
|
||||
metricsList.addAll(subMetricMap.values());
|
||||
groupMetricsList.add(groupMetrics);
|
||||
groupMetricsList.addAll(tpGroupPOMap.values());
|
||||
|
||||
// 记录采集性能
|
||||
groupMetrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, (System.currentTimeMillis() - startTime) / 1000.0f);
|
||||
|
||||
metricsMap.put(groupName, metricsList);
|
||||
metricsMap.put(groupName, groupMetricsList);
|
||||
}
|
||||
|
||||
private String genTopicPartitionGroupKey(String topic, Integer partitionId){
|
||||
return topic + "@" + partitionId;
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric.kafka;
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
@@ -9,6 +9,8 @@ import com.xiaojukeji.know.streaming.km.common.bean.entity.topic.Topic;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionControlItem;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.PartitionMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.EnvUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.partition.PartitionMetricService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.topic.TopicService;
|
||||
@@ -25,8 +27,8 @@ import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemT
|
||||
* @author didi
|
||||
*/
|
||||
@Component
|
||||
public class PartitionMetricCollector extends AbstractKafkaMetricCollector<PartitionMetrics> {
|
||||
protected static final ILog LOGGER = LogFactory.getLog(PartitionMetricCollector.class);
|
||||
public class PartitionMetricCollector extends AbstractMetricCollector<PartitionMetrics> {
|
||||
protected static final ILog LOGGER = LogFactory.getLog("METRIC_LOGGER");
|
||||
|
||||
@Autowired
|
||||
private VersionControlService versionControlService;
|
||||
@@ -38,10 +40,13 @@ public class PartitionMetricCollector extends AbstractKafkaMetricCollector<Parti
|
||||
private TopicService topicService;
|
||||
|
||||
@Override
|
||||
public List<PartitionMetrics> collectKafkaMetrics(ClusterPhy clusterPhy) {
|
||||
public void collectMetrics(ClusterPhy clusterPhy) {
|
||||
Long startTime = System.currentTimeMillis();
|
||||
Long clusterPhyId = clusterPhy.getId();
|
||||
List<Topic> topicList = topicService.listTopicsFromCacheFirst(clusterPhyId);
|
||||
List<VersionControlItem> items = versionControlService.listVersionControlItem(this.getClusterVersion(clusterPhy), collectorType().getCode());
|
||||
List<VersionControlItem> items = versionControlService.listVersionControlItem(clusterPhyId, collectorType().getCode());
|
||||
|
||||
// 获取集群所有分区
|
||||
|
||||
FutureWaitUtil<Void> future = this.getFutureUtilByClusterPhyId(clusterPhyId);
|
||||
|
||||
@@ -50,9 +55,9 @@ public class PartitionMetricCollector extends AbstractKafkaMetricCollector<Parti
|
||||
metricsMap.put(topic.getTopicName(), new ConcurrentHashMap<>());
|
||||
|
||||
future.runnableTask(
|
||||
String.format("class=PartitionMetricCollector||clusterPhyId=%d||topicName=%s", clusterPhyId, topic.getTopicName()),
|
||||
String.format("method=PartitionMetricCollector||clusterPhyId=%d||topicName=%s", clusterPhyId, topic.getTopicName()),
|
||||
30000,
|
||||
() -> this.collectMetrics(clusterPhyId, topic.getTopicName(), metricsMap.get(topic.getTopicName()), items)
|
||||
() -> collectMetrics(clusterPhyId, topic.getTopicName(), metricsMap.get(topic.getTopicName()), items)
|
||||
);
|
||||
}
|
||||
|
||||
@@ -63,7 +68,10 @@ public class PartitionMetricCollector extends AbstractKafkaMetricCollector<Parti
|
||||
|
||||
this.publishMetric(new PartitionMetricEvent(this, metricsList));
|
||||
|
||||
return metricsList;
|
||||
LOGGER.info(
|
||||
"method=PartitionMetricCollector||clusterPhyId={}||startTime={}||costTime={}||msg=collect finished.",
|
||||
clusterPhyId, startTime, System.currentTimeMillis() - startTime
|
||||
);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -101,9 +109,17 @@ public class PartitionMetricCollector extends AbstractKafkaMetricCollector<Parti
|
||||
PartitionMetrics allMetrics = metricsMap.get(subMetrics.getPartitionId());
|
||||
allMetrics.putMetric(subMetrics.getMetrics());
|
||||
}
|
||||
|
||||
if (!EnvUtil.isOnline()) {
|
||||
LOGGER.info(
|
||||
"class=PartitionMetricCollector||method=collectMetrics||clusterPhyId={}||topicName={}||metricName={}||metricValue={}!",
|
||||
clusterPhyId, topicName, v.getName(), ConvertUtil.obj2Json(ret.getData())
|
||||
);
|
||||
}
|
||||
|
||||
} catch (Exception e) {
|
||||
LOGGER.info(
|
||||
"method=collectMetrics||clusterPhyId={}||topicName={}||metricName={}||errMsg=exception",
|
||||
"class=PartitionMetricCollector||method=collectMetrics||clusterPhyId={}||topicName={}||metricName={}||errMsg=exception",
|
||||
clusterPhyId, topicName, v.getName(), e
|
||||
);
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric.kafka;
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric;
|
||||
|
||||
import com.alibaba.fastjson.JSON;
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
|
||||
@@ -10,6 +11,7 @@ import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionContro
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.ReplicaMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.Constant;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.EnvUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.partition.PartitionService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.replica.ReplicaMetricService;
|
||||
@@ -26,8 +28,8 @@ import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemT
|
||||
* @author didi
|
||||
*/
|
||||
@Component
|
||||
public class ReplicaMetricCollector extends AbstractKafkaMetricCollector<ReplicationMetrics> {
|
||||
protected static final ILog LOGGER = LogFactory.getLog(ReplicaMetricCollector.class);
|
||||
public class ReplicaMetricCollector extends AbstractMetricCollector<ReplicationMetrics> {
|
||||
protected static final ILog LOGGER = LogFactory.getLog("METRIC_LOGGER");
|
||||
|
||||
@Autowired
|
||||
private VersionControlService versionControlService;
|
||||
@@ -39,10 +41,12 @@ public class ReplicaMetricCollector extends AbstractKafkaMetricCollector<Replica
|
||||
private PartitionService partitionService;
|
||||
|
||||
@Override
|
||||
public List<ReplicationMetrics> collectKafkaMetrics(ClusterPhy clusterPhy) {
|
||||
public void collectMetrics(ClusterPhy clusterPhy) {
|
||||
Long startTime = System.currentTimeMillis();
|
||||
Long clusterPhyId = clusterPhy.getId();
|
||||
List<Partition> partitions = partitionService.listPartitionFromCacheFirst(clusterPhyId);
|
||||
List<VersionControlItem> items = versionControlService.listVersionControlItem(this.getClusterVersion(clusterPhy), collectorType().getCode());
|
||||
List<VersionControlItem> items = versionControlService.listVersionControlItem(clusterPhyId, collectorType().getCode());
|
||||
|
||||
List<Partition> partitions = partitionService.listPartitionByCluster(clusterPhyId);
|
||||
|
||||
FutureWaitUtil<Void> future = this.getFutureUtilByClusterPhyId(clusterPhyId);
|
||||
|
||||
@@ -50,11 +54,10 @@ public class ReplicaMetricCollector extends AbstractKafkaMetricCollector<Replica
|
||||
for(Partition partition : partitions) {
|
||||
for (Integer brokerId: partition.getAssignReplicaList()) {
|
||||
ReplicationMetrics metrics = new ReplicationMetrics(clusterPhyId, partition.getTopicName(), brokerId, partition.getPartitionId());
|
||||
metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
|
||||
metricsList.add(metrics);
|
||||
|
||||
future.runnableTask(
|
||||
String.format("class=ReplicaMetricCollector||clusterPhyId=%d||brokerId=%d||topicName=%s||partitionId=%d",
|
||||
String.format("method=ReplicaMetricCollector||clusterPhyId=%d||brokerId=%d||topicName=%s||partitionId=%d",
|
||||
clusterPhyId, brokerId, partition.getTopicName(), partition.getPartitionId()),
|
||||
30000,
|
||||
() -> collectMetrics(clusterPhyId, metrics, items)
|
||||
@@ -66,7 +69,8 @@ public class ReplicaMetricCollector extends AbstractKafkaMetricCollector<Replica
|
||||
|
||||
publishMetric(new ReplicaMetricEvent(this, metricsList));
|
||||
|
||||
return metricsList;
|
||||
LOGGER.info("method=ReplicaMetricCollector||clusterPhyId={}||startTime={}||costTime={}||msg=collect finished.",
|
||||
clusterPhyId, startTime, System.currentTimeMillis() - startTime);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -79,13 +83,15 @@ public class ReplicaMetricCollector extends AbstractKafkaMetricCollector<Replica
|
||||
private ReplicationMetrics collectMetrics(Long clusterPhyId, ReplicationMetrics metrics, List<VersionControlItem> items) {
|
||||
long startTime = System.currentTimeMillis();
|
||||
|
||||
metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
|
||||
|
||||
for(VersionControlItem v : items) {
|
||||
try {
|
||||
if (metrics.getMetrics().containsKey(v.getName())) {
|
||||
continue;
|
||||
}
|
||||
|
||||
Result<ReplicationMetrics> ret = replicaMetricService.collectReplicaMetricsFromKafka(
|
||||
Result<ReplicationMetrics> ret = replicaMetricService.collectReplicaMetricsFromKafkaWithCache(
|
||||
clusterPhyId,
|
||||
metrics.getTopic(),
|
||||
metrics.getBrokerId(),
|
||||
@@ -98,11 +104,15 @@ public class ReplicaMetricCollector extends AbstractKafkaMetricCollector<Replica
|
||||
}
|
||||
|
||||
metrics.putMetric(ret.getData().getMetrics());
|
||||
|
||||
if (!EnvUtil.isOnline()) {
|
||||
LOGGER.info("method=ReplicaMetricCollector||clusterPhyId={}||topicName={}||partitionId={}||metricName={}||metricValue={}",
|
||||
clusterPhyId, metrics.getTopic(), metrics.getPartitionId(), v.getName(), JSON.toJSONString(ret.getData().getMetrics()));
|
||||
}
|
||||
|
||||
} catch (Exception e) {
|
||||
LOGGER.error(
|
||||
"method=collectMetrics||clusterPhyId={}||topicName={}||partition={}||metricName={}||errMsg=exception!",
|
||||
clusterPhyId, metrics.getTopic(), metrics.getPartitionId(), v.getName(), e
|
||||
);
|
||||
LOGGER.error("method=ReplicaMetricCollector||clusterPhyId={}||topicName={}||partition={}||metricName={}||errMsg=exception!",
|
||||
clusterPhyId, metrics.getTopic(), metrics.getPartitionId(), v.getName(), e);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric.kafka;
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
@@ -10,6 +10,8 @@ import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionContro
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.TopicMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.Constant;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.EnvUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.topic.TopicMetricService;
|
||||
@@ -29,8 +31,8 @@ import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemT
|
||||
* @author didi
|
||||
*/
|
||||
@Component
|
||||
public class TopicMetricCollector extends AbstractKafkaMetricCollector<TopicMetrics> {
|
||||
protected static final ILog LOGGER = LogFactory.getLog(TopicMetricCollector.class);
|
||||
public class TopicMetricCollector extends AbstractMetricCollector<List<TopicMetrics>> {
|
||||
protected static final ILog LOGGER = LogFactory.getLog("METRIC_LOGGER");
|
||||
|
||||
@Autowired
|
||||
private VersionControlService versionControlService;
|
||||
@@ -44,10 +46,11 @@ public class TopicMetricCollector extends AbstractKafkaMetricCollector<TopicMetr
|
||||
private static final Integer AGG_METRICS_BROKER_ID = -10000;
|
||||
|
||||
@Override
|
||||
public List<TopicMetrics> collectKafkaMetrics(ClusterPhy clusterPhy) {
|
||||
public void collectMetrics(ClusterPhy clusterPhy) {
|
||||
Long startTime = System.currentTimeMillis();
|
||||
Long clusterPhyId = clusterPhy.getId();
|
||||
List<Topic> topics = topicService.listTopicsFromCacheFirst(clusterPhyId);
|
||||
List<VersionControlItem> items = versionControlService.listVersionControlItem(this.getClusterVersion(clusterPhy), collectorType().getCode());
|
||||
List<VersionControlItem> items = versionControlService.listVersionControlItem(clusterPhyId, collectorType().getCode());
|
||||
|
||||
FutureWaitUtil<Void> future = this.getFutureUtilByClusterPhyId(clusterPhyId);
|
||||
|
||||
@@ -61,7 +64,7 @@ public class TopicMetricCollector extends AbstractKafkaMetricCollector<TopicMetr
|
||||
allMetricsMap.put(topic.getTopicName(), metricsMap);
|
||||
|
||||
future.runnableTask(
|
||||
String.format("class=TopicMetricCollector||clusterPhyId=%d||topicName=%s", clusterPhyId, topic.getTopicName()),
|
||||
String.format("method=TopicMetricCollector||clusterPhyId=%d||topicName=%s", clusterPhyId, topic.getTopicName()),
|
||||
30000,
|
||||
() -> collectMetrics(clusterPhyId, topic.getTopicName(), metricsMap, items)
|
||||
);
|
||||
@@ -74,7 +77,8 @@ public class TopicMetricCollector extends AbstractKafkaMetricCollector<TopicMetr
|
||||
|
||||
this.publishMetric(new TopicMetricEvent(this, metricsList));
|
||||
|
||||
return metricsList;
|
||||
LOGGER.info("method=TopicMetricCollector||clusterPhyId={}||startTime={}||costTime={}||msg=collect finished.",
|
||||
clusterPhyId, startTime, System.currentTimeMillis() - startTime);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -114,9 +118,14 @@ public class TopicMetricCollector extends AbstractKafkaMetricCollector<TopicMetr
|
||||
metricsMap.get(metrics.getBrokerId()).putMetric(metrics.getMetrics());
|
||||
}
|
||||
});
|
||||
|
||||
if (!EnvUtil.isOnline()) {
|
||||
LOGGER.info("method=TopicMetricCollector||clusterPhyId={}||topicName={}||metricName={}||metricValue={}.",
|
||||
clusterPhyId, topicName, v.getName(), ConvertUtil.obj2Json(ret.getData())
|
||||
);
|
||||
}
|
||||
} catch (Exception e) {
|
||||
LOGGER.error(
|
||||
"method=collectMetrics||clusterPhyId={}||topicName={}||metricName={}||errMsg=exception!",
|
||||
LOGGER.error("method=TopicMetricCollector||clusterPhyId={}||topicName={}||metricName={}||errMsg=exception!",
|
||||
clusterPhyId, topicName, v.getName(), e
|
||||
);
|
||||
}
|
||||
@@ -1,50 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric.connect;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.collector.metric.AbstractMetricCollector;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.ConnectCluster;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.LoggerUtil;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.connect.cluster.ConnectClusterService;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* @author didi
|
||||
*/
|
||||
public abstract class AbstractConnectMetricCollector<M> extends AbstractMetricCollector<M, ConnectCluster> {
|
||||
private static final ILog LOGGER = LogFactory.getLog(AbstractConnectMetricCollector.class);
|
||||
|
||||
protected static final ILog METRIC_COLLECTED_LOGGER = LoggerUtil.getMetricCollectedLogger();
|
||||
|
||||
@Autowired
|
||||
private ConnectClusterService connectClusterService;
|
||||
|
||||
public abstract List<M> collectConnectMetrics(ConnectCluster connectCluster);
|
||||
|
||||
@Override
|
||||
public String getClusterVersion(ConnectCluster connectCluster){
|
||||
return connectClusterService.getClusterVersion(connectCluster.getId());
|
||||
}
|
||||
|
||||
@Override
|
||||
public void collectMetrics(ConnectCluster connectCluster) {
|
||||
long startTime = System.currentTimeMillis();
|
||||
|
||||
// 采集指标
|
||||
List<M> metricsList = this.collectConnectMetrics(connectCluster);
|
||||
|
||||
// 输出耗时信息
|
||||
LOGGER.info(
|
||||
"metricType={}||connectClusterId={}||costTimeUnitMs={}",
|
||||
this.collectorType().getMessage(), connectCluster.getId(), System.currentTimeMillis() - startTime
|
||||
);
|
||||
|
||||
// 输出采集到的指标信息
|
||||
METRIC_COLLECTED_LOGGER.debug("metricType={}||connectClusterId={}||metrics={}!",
|
||||
this.collectorType().getMessage(), connectCluster.getId(), ConvertUtil.obj2Json(metricsList)
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -1,83 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric.connect;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.ConnectCluster;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.connect.ConnectClusterMetrics;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionControlItem;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.connect.ConnectClusterMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.Constant;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.connect.cluster.ConnectClusterMetricService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.VersionControlService;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum.METRIC_CONNECT_CLUSTER;
|
||||
|
||||
/**
|
||||
* @author didi
|
||||
*/
|
||||
@Component
|
||||
public class ConnectClusterMetricCollector extends AbstractConnectMetricCollector<ConnectClusterMetrics> {
|
||||
protected static final ILog LOGGER = LogFactory.getLog(ConnectClusterMetricCollector.class);
|
||||
|
||||
@Autowired
|
||||
private VersionControlService versionControlService;
|
||||
|
||||
@Autowired
|
||||
private ConnectClusterMetricService connectClusterMetricService;
|
||||
|
||||
@Override
|
||||
public List<ConnectClusterMetrics> collectConnectMetrics(ConnectCluster connectCluster) {
|
||||
Long startTime = System.currentTimeMillis();
|
||||
Long clusterPhyId = connectCluster.getKafkaClusterPhyId();
|
||||
Long connectClusterId = connectCluster.getId();
|
||||
|
||||
ConnectClusterMetrics metrics = new ConnectClusterMetrics(clusterPhyId, connectClusterId);
|
||||
metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
|
||||
List<VersionControlItem> items = versionControlService.listVersionControlItem(getClusterVersion(connectCluster), collectorType().getCode());
|
||||
FutureWaitUtil<Void> future = this.getFutureUtilByClusterPhyId(connectClusterId);
|
||||
|
||||
for (VersionControlItem item : items) {
|
||||
future.runnableTask(
|
||||
String.format("class=ConnectClusterMetricCollector||connectClusterId=%d||metricName=%s", connectClusterId, item.getName()),
|
||||
30000,
|
||||
() -> {
|
||||
try {
|
||||
Result<ConnectClusterMetrics> ret = connectClusterMetricService.collectConnectClusterMetricsFromKafka(connectClusterId, item.getName());
|
||||
if (null == ret || !ret.hasData()) {
|
||||
return null;
|
||||
}
|
||||
metrics.putMetric(ret.getData().getMetrics());
|
||||
|
||||
} catch (Exception e) {
|
||||
LOGGER.error(
|
||||
"method=collectConnectMetrics||connectClusterId={}||metricName={}||errMsg=exception!",
|
||||
connectClusterId, item.getName(), e
|
||||
);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
future.waitExecute(30000);
|
||||
|
||||
metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, (System.currentTimeMillis() - startTime) / 1000.0f);
|
||||
|
||||
this.publishMetric(new ConnectClusterMetricEvent(this, Collections.singletonList(metrics)));
|
||||
|
||||
return Collections.singletonList(metrics);
|
||||
}
|
||||
|
||||
@Override
|
||||
public VersionItemTypeEnum collectorType() {
|
||||
return METRIC_CONNECT_CLUSTER;
|
||||
}
|
||||
}
|
||||
@@ -1,102 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric.connect;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.connect.ConnectCluster;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.connect.ConnectorMetrics;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionControlItem;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.connect.ConnectorMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.Constant;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.connect.ConnectorTypeEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.FutureWaitUtil;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.connect.connector.ConnectorMetricService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.connect.connector.ConnectorService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.VersionControlService;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum.METRIC_CONNECT_CONNECTOR;
|
||||
|
||||
/**
|
||||
* @author didi
|
||||
*/
|
||||
@Component
|
||||
public class ConnectConnectorMetricCollector extends AbstractConnectMetricCollector<ConnectorMetrics> {
|
||||
protected static final ILog LOGGER = LogFactory.getLog(ConnectConnectorMetricCollector.class);
|
||||
|
||||
@Autowired
|
||||
private VersionControlService versionControlService;
|
||||
|
||||
@Autowired
|
||||
private ConnectorService connectorService;
|
||||
|
||||
@Autowired
|
||||
private ConnectorMetricService connectorMetricService;
|
||||
|
||||
@Override
|
||||
public List<ConnectorMetrics> collectConnectMetrics(ConnectCluster connectCluster) {
|
||||
Long clusterPhyId = connectCluster.getKafkaClusterPhyId();
|
||||
Long connectClusterId = connectCluster.getId();
|
||||
|
||||
List<VersionControlItem> items = versionControlService.listVersionControlItem(this.getClusterVersion(connectCluster), collectorType().getCode());
|
||||
Result<List<String>> connectorList = connectorService.listConnectorsFromCluster(connectClusterId);
|
||||
|
||||
FutureWaitUtil<Void> future = this.getFutureUtilByClusterPhyId(connectClusterId);
|
||||
|
||||
List<ConnectorMetrics> metricsList = new ArrayList<>();
|
||||
for (String connectorName : connectorList.getData()) {
|
||||
ConnectorMetrics metrics = new ConnectorMetrics(connectClusterId, connectorName);
|
||||
metrics.setClusterPhyId(clusterPhyId);
|
||||
|
||||
metricsList.add(metrics);
|
||||
future.runnableTask(
|
||||
String.format("class=ConnectConnectorMetricCollector||connectClusterId=%d||connectorName=%s", connectClusterId, connectorName),
|
||||
30000,
|
||||
() -> collectMetrics(connectClusterId, connectorName, metrics, items)
|
||||
);
|
||||
}
|
||||
future.waitResult(30000);
|
||||
|
||||
this.publishMetric(new ConnectorMetricEvent(this, metricsList));
|
||||
|
||||
return metricsList;
|
||||
}
|
||||
|
||||
@Override
|
||||
public VersionItemTypeEnum collectorType() {
|
||||
return METRIC_CONNECT_CONNECTOR;
|
||||
}
|
||||
|
||||
/**************************************************** private method ****************************************************/
|
||||
|
||||
private void collectMetrics(Long connectClusterId, String connectorName, ConnectorMetrics metrics, List<VersionControlItem> items) {
|
||||
long startTime = System.currentTimeMillis();
|
||||
ConnectorTypeEnum connectorType = connectorService.getConnectorType(connectClusterId, connectorName);
|
||||
|
||||
metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
|
||||
|
||||
for (VersionControlItem v : items) {
|
||||
try {
|
||||
Result<ConnectorMetrics> ret = connectorMetricService.collectConnectClusterMetricsFromKafka(connectClusterId, connectorName, v.getName(), connectorType);
|
||||
if (null == ret || ret.failed() || null == ret.getData()) {
|
||||
continue;
|
||||
}
|
||||
|
||||
metrics.putMetric(ret.getData().getMetrics());
|
||||
} catch (Exception e) {
|
||||
LOGGER.error(
|
||||
"method=collectMetrics||connectClusterId={}||connectorName={}||metric={}||errMsg=exception!",
|
||||
connectClusterId, connectorName, v.getName(), e
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// 记录采集性能
|
||||
metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, (System.currentTimeMillis() - startTime) / 1000.0f);
|
||||
}
|
||||
}
|
||||
@@ -1,50 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric.kafka;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.collector.metric.AbstractMetricCollector;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.LoggerUtil;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.cluster.ClusterPhyService;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* @author didi
|
||||
*/
|
||||
public abstract class AbstractKafkaMetricCollector<M> extends AbstractMetricCollector<M, ClusterPhy> {
|
||||
private static final ILog LOGGER = LogFactory.getLog(AbstractMetricCollector.class);
|
||||
|
||||
protected static final ILog METRIC_COLLECTED_LOGGER = LoggerUtil.getMetricCollectedLogger();
|
||||
|
||||
@Autowired
|
||||
private ClusterPhyService clusterPhyService;
|
||||
|
||||
public abstract List<M> collectKafkaMetrics(ClusterPhy clusterPhy);
|
||||
|
||||
@Override
|
||||
public String getClusterVersion(ClusterPhy clusterPhy){
|
||||
return clusterPhyService.getVersionFromCacheFirst(clusterPhy.getId());
|
||||
}
|
||||
|
||||
@Override
|
||||
public void collectMetrics(ClusterPhy clusterPhy) {
|
||||
long startTime = System.currentTimeMillis();
|
||||
|
||||
// 采集指标
|
||||
List<M> metricsList = this.collectKafkaMetrics(clusterPhy);
|
||||
|
||||
// 输出耗时信息
|
||||
LOGGER.info(
|
||||
"metricType={}||clusterPhyId={}||costTimeUnitMs={}",
|
||||
this.collectorType().getMessage(), clusterPhy.getId(), System.currentTimeMillis() - startTime
|
||||
);
|
||||
|
||||
// 输出采集到的指标信息
|
||||
METRIC_COLLECTED_LOGGER.debug("metricType={}||clusterPhyId={}||metrics={}!",
|
||||
this.collectorType().getMessage(), clusterPhy.getId(), ConvertUtil.obj2Json(metricsList)
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -1,111 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.metric.kafka;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.cluster.ClusterPhy;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.config.ZKConfig;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.kafkacontroller.KafkaController;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.param.metric.ZookeeperMetricParam;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.result.Result;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.version.VersionControlItem;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.Tuple;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ValidateUtils;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.ZookeeperMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.Constant;
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.zookeeper.ZookeeperInfo;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.metrics.ZookeeperMetrics;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.kafkacontroller.KafkaControllerService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.version.VersionControlService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.zookeeper.ZookeeperMetricService;
|
||||
import com.xiaojukeji.know.streaming.km.core.service.zookeeper.ZookeeperService;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.common.enums.version.VersionItemTypeEnum.METRIC_ZOOKEEPER;
|
||||
|
||||
/**
|
||||
* @author didi
|
||||
*/
|
||||
@Component
|
||||
public class ZookeeperMetricCollector extends AbstractKafkaMetricCollector<ZookeeperMetrics> {
|
||||
protected static final ILog LOGGER = LogFactory.getLog(ZookeeperMetricCollector.class);
|
||||
|
||||
@Autowired
|
||||
private VersionControlService versionControlService;
|
||||
|
||||
@Autowired
|
||||
private ZookeeperMetricService zookeeperMetricService;
|
||||
|
||||
@Autowired
|
||||
private ZookeeperService zookeeperService;
|
||||
|
||||
@Autowired
|
||||
private KafkaControllerService kafkaControllerService;
|
||||
|
||||
@Override
|
||||
public List<ZookeeperMetrics> collectKafkaMetrics(ClusterPhy clusterPhy) {
|
||||
Long startTime = System.currentTimeMillis();
|
||||
Long clusterPhyId = clusterPhy.getId();
|
||||
List<VersionControlItem> items = versionControlService.listVersionControlItem(this.getClusterVersion(clusterPhy), collectorType().getCode());
|
||||
List<ZookeeperInfo> aliveZKList = zookeeperService.listFromDBByCluster(clusterPhyId)
|
||||
.stream()
|
||||
.filter(elem -> Constant.ALIVE.equals(elem.getStatus()))
|
||||
.collect(Collectors.toList());
|
||||
KafkaController kafkaController = kafkaControllerService.getKafkaControllerFromDB(clusterPhyId);
|
||||
|
||||
ZookeeperMetrics metrics = ZookeeperMetrics.initWithMetric(clusterPhyId, Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, Constant.COLLECT_METRICS_ERROR_COST_TIME);
|
||||
if (ValidateUtils.isEmptyList(aliveZKList)) {
|
||||
// 没有存活的ZK时，发布事件，然后直接返回
|
||||
publishMetric(new ZookeeperMetricEvent(this, Collections.singletonList(metrics)));
|
||||
return Collections.singletonList(metrics);
|
||||
}
|
||||
|
||||
// 构造参数
|
||||
ZookeeperMetricParam param = new ZookeeperMetricParam(
|
||||
clusterPhyId,
|
||||
aliveZKList.stream().map(elem -> new Tuple<String, Integer>(elem.getHost(), elem.getPort())).collect(Collectors.toList()),
|
||||
ConvertUtil.str2ObjByJson(clusterPhy.getZkProperties(), ZKConfig.class),
|
||||
kafkaController == null? Constant.INVALID_CODE: kafkaController.getBrokerId(),
|
||||
null
|
||||
);
|
||||
|
||||
for(VersionControlItem v : items) {
|
||||
try {
|
||||
if(null != metrics.getMetrics().get(v.getName())) {
|
||||
continue;
|
||||
}
|
||||
|
||||
param.setMetricName(v.getName());
|
||||
|
||||
Result<ZookeeperMetrics> ret = zookeeperMetricService.collectMetricsFromZookeeper(param);
|
||||
if(null == ret || ret.failed() || null == ret.getData()){
|
||||
continue;
|
||||
}
|
||||
|
||||
metrics.putMetric(ret.getData().getMetrics());
|
||||
} catch (Exception e){
|
||||
LOGGER.error(
|
||||
"method=collectMetrics||clusterPhyId={}||metricName={}||errMsg=exception!",
|
||||
clusterPhyId, v.getName(), e
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
metrics.putMetric(Constant.COLLECT_METRICS_COST_TIME_METRICS_NAME, (System.currentTimeMillis() - startTime) / 1000.0f);
|
||||
|
||||
this.publishMetric(new ZookeeperMetricEvent(this, Collections.singletonList(metrics)));
|
||||
|
||||
return Collections.singletonList(metrics);
|
||||
}
|
||||
|
||||
@Override
|
||||
public VersionItemTypeEnum collectorType() {
|
||||
return METRIC_ZOOKEEPER;
|
||||
}
|
||||
}
|
||||
@@ -237,7 +237,7 @@ public class CollectThreadPoolService {
|
||||
private synchronized FutureWaitUtil<Void> closeOldAndCreateNew(Long shardId) {
|
||||
// 新的
|
||||
FutureWaitUtil<Void> newFutureUtil = FutureWaitUtil.init(
|
||||
"MetricCollect-Shard-" + shardId,
|
||||
"CollectorMetricsFutureUtil-Shard-" + shardId,
|
||||
this.futureUtilThreadNum,
|
||||
this.futureUtilThreadNum,
|
||||
this.futureUtilQueueSize
|
||||
|
||||
@@ -3,47 +3,67 @@ package com.xiaojukeji.know.streaming.km.collector.sink;
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.po.BaseESPO;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.FutureUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.EnvUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.NamedThreadFactory;
|
||||
import com.xiaojukeji.know.streaming.km.persistence.es.dao.BaseMetricESDAO;
|
||||
import org.apache.commons.collections.CollectionUtils;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Objects;
|
||||
import java.util.concurrent.LinkedBlockingDeque;
|
||||
import java.util.concurrent.ThreadPoolExecutor;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
public abstract class AbstractMetricESSender {
|
||||
private static final ILog LOGGER = LogFactory.getLog(AbstractMetricESSender.class);
|
||||
protected static final ILog LOGGER = LogFactory.getLog("METRIC_LOGGER");
|
||||
|
||||
private static final int THRESHOLD = 100;
|
||||
private static final int THRESHOLD = 100;
|
||||
|
||||
private static final FutureUtil<Void> esExecutor = FutureUtil.init(
|
||||
"MetricsESSender",
|
||||
private static final ThreadPoolExecutor esExecutor = new ThreadPoolExecutor(
|
||||
10,
|
||||
20,
|
||||
10000
|
||||
6000,
|
||||
TimeUnit.MILLISECONDS,
|
||||
new LinkedBlockingDeque<>(1000),
|
||||
new NamedThreadFactory("KM-Collect-MetricESSender-ES"),
|
||||
(r, e) -> LOGGER.warn("class=MetricESSender||msg=KM-Collect-MetricESSender-ES Deque is blocked, taskCount:{}" + e.getTaskCount())
|
||||
);
|
||||
|
||||
/**
|
||||
* 根据不同监控维度来发送
|
||||
*/
|
||||
protected boolean send2es(String index, List<? extends BaseESPO> statsList) {
|
||||
LOGGER.info("method=send2es||indexName={}||metricsSize={}||msg=send metrics to es", index, statsList.size());
|
||||
|
||||
protected boolean send2es(String index, List<? extends BaseESPO> statsList){
|
||||
if (CollectionUtils.isEmpty(statsList)) {
|
||||
return true;
|
||||
}
|
||||
|
||||
if (!EnvUtil.isOnline()) {
|
||||
LOGGER.info("class=MetricESSender||method=send2es||ariusStats={}||size={}",
|
||||
index, statsList.size());
|
||||
}
|
||||
|
||||
BaseMetricESDAO baseMetricESDao = BaseMetricESDAO.getByStatsType(index);
|
||||
if (Objects.isNull(baseMetricESDao)) {
|
||||
LOGGER.error("method=send2es||indexName={}||errMsg=find dao failed", index);
|
||||
if (Objects.isNull( baseMetricESDao )) {
|
||||
LOGGER.error("class=MetricESSender||method=send2es||errMsg=fail to find {}", index);
|
||||
return false;
|
||||
}
|
||||
|
||||
for (int i = 0; i < statsList.size(); i += THRESHOLD) {
|
||||
final int idxStart = i;
|
||||
int size = statsList.size();
|
||||
int num = (size) % THRESHOLD == 0 ? (size / THRESHOLD) : (size / THRESHOLD + 1);
|
||||
|
||||
// 异步发送
|
||||
esExecutor.submitTask(
|
||||
() -> baseMetricESDao.batchInsertStats(statsList.subList(idxStart, Math.min(idxStart + THRESHOLD, statsList.size())))
|
||||
if (size < THRESHOLD) {
|
||||
esExecutor.execute(
|
||||
() -> baseMetricESDao.batchInsertStats(statsList)
|
||||
);
|
||||
return true;
|
||||
}
|
||||
|
||||
for (int i = 1; i < num + 1; i++) {
|
||||
int end = (i * THRESHOLD) > size ? size : (i * THRESHOLD);
|
||||
int start = (i - 1) * THRESHOLD;
|
||||
|
||||
esExecutor.execute(
|
||||
() -> baseMetricESDao.batchInsertStats(statsList.subList(start, end))
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.sink.kafka;
|
||||
package com.xiaojukeji.know.streaming.km.collector.sink;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.BrokerMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.BrokerMetricPO;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
@@ -11,15 +10,15 @@ import org.springframework.stereotype.Component;
|
||||
|
||||
import javax.annotation.PostConstruct;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.BROKER_INDEX;
|
||||
import static com.xiaojukeji.know.streaming.km.common.constant.ESIndexConstant.BROKER_INDEX;
|
||||
|
||||
@Component
|
||||
public class BrokerMetricESSender extends AbstractMetricESSender implements ApplicationListener<BrokerMetricEvent> {
|
||||
private static final ILog LOGGER = LogFactory.getLog(BrokerMetricESSender.class);
|
||||
protected static final ILog LOGGER = LogFactory.getLog("METRIC_LOGGER");
|
||||
|
||||
@PostConstruct
|
||||
public void init(){
|
||||
LOGGER.info("method=init||msg=init finished");
|
||||
LOGGER.info("class=BrokerMetricESSender||method=init||msg=init finished");
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -1,8 +1,7 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.sink.kafka;
|
||||
package com.xiaojukeji.know.streaming.km.collector.sink;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.ClusterMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.ClusterMetricPO;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
@@ -11,15 +10,16 @@ import org.springframework.stereotype.Component;
|
||||
|
||||
import javax.annotation.PostConstruct;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.CLUSTER_INDEX;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.common.constant.ESIndexConstant.CLUSTER_INDEX;
|
||||
|
||||
@Component
|
||||
public class ClusterMetricESSender extends AbstractMetricESSender implements ApplicationListener<ClusterMetricEvent> {
|
||||
private static final ILog LOGGER = LogFactory.getLog(ClusterMetricESSender.class);
|
||||
protected static final ILog LOGGER = LogFactory.getLog("METRIC_LOGGER");
|
||||
|
||||
@PostConstruct
|
||||
public void init(){
|
||||
LOGGER.info("method=init||msg=init finished");
|
||||
LOGGER.info("class=ClusterMetricESSender||method=init||msg=init finished");
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -1,8 +1,7 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.sink.kafka;
|
||||
package com.xiaojukeji.know.streaming.km.collector.sink;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.GroupMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.GroupMetricPO;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
@@ -11,15 +10,16 @@ import org.springframework.stereotype.Component;
|
||||
|
||||
import javax.annotation.PostConstruct;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.GROUP_INDEX;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.common.constant.ESIndexConstant.GROUP_INDEX;
|
||||
|
||||
@Component
|
||||
public class GroupMetricESSender extends AbstractMetricESSender implements ApplicationListener<GroupMetricEvent> {
|
||||
private static final ILog LOGGER = LogFactory.getLog(GroupMetricESSender.class);
|
||||
protected static final ILog LOGGER = LogFactory.getLog("METRIC_LOGGER");
|
||||
|
||||
@PostConstruct
|
||||
public void init(){
|
||||
LOGGER.info("method=init||msg=init finished");
|
||||
LOGGER.info("class=GroupMetricESSender||method=init||msg=init finished");
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -1,8 +1,7 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.sink.kafka;
|
||||
package com.xiaojukeji.know.streaming.km.collector.sink;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.PartitionMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.PartitionMetricPO;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
@@ -11,15 +10,15 @@ import org.springframework.stereotype.Component;
|
||||
|
||||
import javax.annotation.PostConstruct;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.PARTITION_INDEX;
|
||||
import static com.xiaojukeji.know.streaming.km.common.constant.ESIndexConstant.PARTITION_INDEX;
|
||||
|
||||
@Component
|
||||
public class PartitionMetricESSender extends AbstractMetricESSender implements ApplicationListener<PartitionMetricEvent> {
|
||||
private static final ILog LOGGER = LogFactory.getLog(PartitionMetricESSender.class);
|
||||
protected static final ILog LOGGER = LogFactory.getLog("METRIC_LOGGER");
|
||||
|
||||
@PostConstruct
|
||||
public void init(){
|
||||
LOGGER.info("method=init||msg=init finished");
|
||||
LOGGER.info("class=PartitionMetricESSender||method=init||msg=init finished");
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -1,8 +1,7 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.sink.kafka;
|
||||
package com.xiaojukeji.know.streaming.km.collector.sink;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.ReplicaMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.ReplicationMetricPO;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
@@ -11,15 +10,15 @@ import org.springframework.stereotype.Component;
|
||||
|
||||
import javax.annotation.PostConstruct;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.REPLICATION_INDEX;
|
||||
import static com.xiaojukeji.know.streaming.km.common.constant.ESIndexConstant.REPLICATION_INDEX;
|
||||
|
||||
@Component
|
||||
public class ReplicaMetricESSender extends AbstractMetricESSender implements ApplicationListener<ReplicaMetricEvent> {
|
||||
private static final ILog LOGGER = LogFactory.getLog(ReplicaMetricESSender.class);
|
||||
protected static final ILog LOGGER = LogFactory.getLog("METRIC_LOGGER");
|
||||
|
||||
@PostConstruct
|
||||
public void init(){
|
||||
LOGGER.info("method=init||msg=init finished");
|
||||
LOGGER.info("class=GroupMetricESSender||method=init||msg=init finished");
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -1,8 +1,7 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.sink.kafka;
|
||||
package com.xiaojukeji.know.streaming.km.collector.sink;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.*;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.*;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
@@ -11,15 +10,16 @@ import org.springframework.stereotype.Component;
|
||||
|
||||
import javax.annotation.PostConstruct;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.TOPIC_INDEX;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.common.constant.ESIndexConstant.TOPIC_INDEX;
|
||||
|
||||
@Component
|
||||
public class TopicMetricESSender extends AbstractMetricESSender implements ApplicationListener<TopicMetricEvent> {
|
||||
private static final ILog LOGGER = LogFactory.getLog(TopicMetricESSender.class);
|
||||
protected static final ILog LOGGER = LogFactory.getLog("METRIC_LOGGER");
|
||||
|
||||
@PostConstruct
|
||||
public void init(){
|
||||
LOGGER.info("method=init||msg=init finished");
|
||||
LOGGER.info("class=TopicMetricESSender||method=init||msg=init finished");
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -1,33 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.sink.connect;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.connect.ConnectClusterMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.connect.ConnectClusterMetricPO;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
import org.springframework.context.ApplicationListener;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import javax.annotation.PostConstruct;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.CONNECT_CLUSTER_INDEX;
|
||||
|
||||
/**
|
||||
* @author wyb
|
||||
* @date 2022/11/7
|
||||
*/
|
||||
@Component
|
||||
public class ConnectClusterMetricESSender extends AbstractMetricESSender implements ApplicationListener<ConnectClusterMetricEvent> {
|
||||
protected static final ILog LOGGER = LogFactory.getLog(ConnectClusterMetricESSender.class);
|
||||
|
||||
@PostConstruct
|
||||
public void init(){
|
||||
LOGGER.info("class=ConnectClusterMetricESSender||method=init||msg=init finished");
|
||||
}
|
||||
|
||||
@Override
|
||||
public void onApplicationEvent(ConnectClusterMetricEvent event) {
|
||||
send2es(CONNECT_CLUSTER_INDEX, ConvertUtil.list2List(event.getConnectClusterMetrics(), ConnectClusterMetricPO.class));
|
||||
}
|
||||
}
|
||||
@@ -1,33 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.sink.connect;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.connect.ConnectorMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.connect.ConnectorMetricPO;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
import org.springframework.context.ApplicationListener;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import javax.annotation.PostConstruct;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.CONNECT_CONNECTOR_INDEX;
|
||||
|
||||
/**
|
||||
* @author wyb
|
||||
* @date 2022/11/7
|
||||
*/
|
||||
@Component
|
||||
public class ConnectorMetricESSender extends AbstractMetricESSender implements ApplicationListener<ConnectorMetricEvent> {
|
||||
protected static final ILog LOGGER = LogFactory.getLog(ConnectorMetricESSender.class);
|
||||
|
||||
@PostConstruct
|
||||
public void init(){
|
||||
LOGGER.info("class=ConnectorMetricESSender||method=init||msg=init finished");
|
||||
}
|
||||
|
||||
@Override
|
||||
public void onApplicationEvent(ConnectorMetricEvent event) {
|
||||
send2es(CONNECT_CONNECTOR_INDEX, ConvertUtil.list2List(event.getConnectorMetricsList(), ConnectorMetricPO.class));
|
||||
}
|
||||
}
|
||||
@@ -1,29 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.collector.sink.kafka;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.collector.sink.AbstractMetricESSender;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.ConvertUtil;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.event.metric.ZookeeperMetricEvent;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.po.metrice.ZookeeperMetricPO;
|
||||
import org.springframework.context.ApplicationListener;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import javax.annotation.PostConstruct;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.persistence.es.template.TemplateConstant.ZOOKEEPER_INDEX;
|
||||
|
||||
@Component
|
||||
public class ZookeeperMetricESSender extends AbstractMetricESSender implements ApplicationListener<ZookeeperMetricEvent> {
|
||||
private static final ILog LOGGER = LogFactory.getLog(ZookeeperMetricESSender.class);
|
||||
|
||||
@PostConstruct
|
||||
public void init(){
|
||||
LOGGER.info("method=init||msg=init finished");
|
||||
}
|
||||
|
||||
@Override
|
||||
public void onApplicationEvent(ZookeeperMetricEvent event) {
|
||||
send2es(ZOOKEEPER_INDEX, ConvertUtil.list2List(event.getZookeeperMetrics(), ZookeeperMetricPO.class));
|
||||
}
|
||||
}
|
||||
@@ -127,9 +127,5 @@
|
||||
<groupId>org.apache.kafka</groupId>
|
||||
<artifactId>kafka_2.13</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.apache.kafka</groupId>
|
||||
<artifactId>connect-runtime</artifactId>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</project>
|
||||
@@ -1,28 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.dto.cluster;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.MetricDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationSortDTO;
|
||||
import io.swagger.annotations.ApiModelProperty;
|
||||
import lombok.Data;
|
||||
|
||||
import javax.validation.constraints.NotNull;
|
||||
import java.util.List;
|
||||
|
||||
|
||||
/**
|
||||
* @author zengqiao
|
||||
* @date 22/02/24
|
||||
*/
|
||||
@Data
|
||||
public class ClusterConnectorsOverviewDTO extends PaginationSortDTO {
|
||||
@NotNull(message = "latestMetricNames不允许为空")
|
||||
@ApiModelProperty("需要指标点的信息")
|
||||
private List<String> latestMetricNames;
|
||||
|
||||
@NotNull(message = "metricLines不允许为空")
|
||||
@ApiModelProperty("需要指标曲线的信息")
|
||||
private MetricDTO metricLines;
|
||||
|
||||
@ApiModelProperty("需要排序的指标名称列表,比较第一个不为空的metric")
|
||||
private List<String> sortMetricNameList;
|
||||
}
|
||||
@@ -1,18 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.dto.cluster;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationBaseDTO;
|
||||
import io.swagger.annotations.ApiModelProperty;
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @author wyb
|
||||
* @date 2022/10/17
|
||||
*/
|
||||
@Data
|
||||
public class ClusterGroupSummaryDTO extends PaginationBaseDTO {
|
||||
@ApiModelProperty("查找该Topic")
|
||||
private String searchTopicName;
|
||||
|
||||
@ApiModelProperty("查找该Group")
|
||||
private String searchGroupName;
|
||||
}
|
||||
@@ -1,13 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.dto.cluster;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.pagination.PaginationBaseDTO;
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @author wyc
|
||||
* @date 2022/9/23
|
||||
*/
|
||||
@Data
|
||||
public class ClusterZookeepersOverviewDTO extends PaginationBaseDTO {
|
||||
|
||||
}
|
||||
@@ -1,32 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.dto.connect;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.BaseDTO;
|
||||
import io.swagger.annotations.ApiModel;
|
||||
import io.swagger.annotations.ApiModelProperty;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
import javax.validation.constraints.NotBlank;
|
||||
import javax.validation.constraints.NotNull;
|
||||
|
||||
/**
|
||||
* @author zengqiao
|
||||
* @date 2022-10-17
|
||||
*/
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
@ApiModel(description = "集群Connector")
|
||||
public class ClusterConnectorDTO extends BaseDTO {
|
||||
@NotNull(message = "connectClusterId不允许为空")
|
||||
@ApiModelProperty(value = "Connector集群ID", example = "1")
|
||||
private Long connectClusterId;
|
||||
|
||||
@NotBlank(message = "name不允许为空串")
|
||||
@ApiModelProperty(value = "Connector名称", example = "know-streaming-connector")
|
||||
private String connectorName;
|
||||
|
||||
public ClusterConnectorDTO(Long connectClusterId, String connectorName) {
|
||||
this.connectClusterId = connectClusterId;
|
||||
this.connectorName = connectorName;
|
||||
}
|
||||
}
|
||||
@@ -1,29 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.dto.connect.cluster;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.BaseDTO;
|
||||
import io.swagger.annotations.ApiModel;
|
||||
import io.swagger.annotations.ApiModelProperty;
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @author zengqiao
|
||||
* @date 2022-10-17
|
||||
*/
|
||||
@Data
|
||||
@ApiModel(description = "集群Connector")
|
||||
public class ConnectClusterDTO extends BaseDTO {
|
||||
@ApiModelProperty(value = "Connect集群ID", example = "1")
|
||||
private Long id;
|
||||
|
||||
@ApiModelProperty(value = "Connect集群名称", example = "know-streaming")
|
||||
private String name;
|
||||
|
||||
@ApiModelProperty(value = "Connect集群URL", example = "http://127.0.0.1:8080")
|
||||
private String clusterUrl;
|
||||
|
||||
@ApiModelProperty(value = "Connect集群版本", example = "2.5.1")
|
||||
private String version;
|
||||
|
||||
@ApiModelProperty(value = "JMX配置", example = "")
|
||||
private String jmxProperties;
|
||||
}
|
||||
@@ -1,20 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.ClusterConnectorDTO;
|
||||
import io.swagger.annotations.ApiModel;
|
||||
import io.swagger.annotations.ApiModelProperty;
|
||||
import lombok.Data;
|
||||
|
||||
import javax.validation.constraints.NotBlank;
|
||||
|
||||
/**
|
||||
* @author zengqiao
|
||||
* @date 2022-10-17
|
||||
*/
|
||||
@Data
|
||||
@ApiModel(description = "操作Connector")
|
||||
public class ConnectorActionDTO extends ClusterConnectorDTO {
|
||||
@NotBlank(message = "action不允许为空串")
|
||||
@ApiModelProperty(value = "Connector名称", example = "stop|restart|resume")
|
||||
private String action;
|
||||
}
|
||||
@@ -1,21 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.ClusterConnectorDTO;
|
||||
import io.swagger.annotations.ApiModel;
|
||||
import io.swagger.annotations.ApiModelProperty;
|
||||
import lombok.Data;
|
||||
|
||||
import javax.validation.constraints.NotNull;
|
||||
import java.util.Properties;
|
||||
|
||||
/**
|
||||
* @author zengqiao
|
||||
* @date 2022-10-17
|
||||
*/
|
||||
@Data
|
||||
@ApiModel(description = "修改Connector配置")
|
||||
public class ConnectorConfigModifyDTO extends ClusterConnectorDTO {
|
||||
@NotNull(message = "configs不允许为空")
|
||||
@ApiModelProperty(value = "配置", example = "")
|
||||
private Properties configs;
|
||||
}
|
||||
@@ -1,21 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.ClusterConnectorDTO;
|
||||
import io.swagger.annotations.ApiModel;
|
||||
import io.swagger.annotations.ApiModelProperty;
|
||||
import lombok.Data;
|
||||
|
||||
import javax.validation.constraints.NotNull;
|
||||
import java.util.Properties;
|
||||
|
||||
/**
|
||||
* @author zengqiao
|
||||
* @date 2022-10-17
|
||||
*/
|
||||
@Data
|
||||
@ApiModel(description = "创建Connector")
|
||||
public class ConnectorCreateDTO extends ClusterConnectorDTO {
|
||||
@NotNull(message = "configs不允许为空")
|
||||
@ApiModelProperty(value = "配置", example = "")
|
||||
private Properties configs;
|
||||
}
|
||||
@@ -1,14 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.ClusterConnectorDTO;
|
||||
import io.swagger.annotations.ApiModel;
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @author zengqiao
|
||||
* @date 2022-10-17
|
||||
*/
|
||||
@Data
|
||||
@ApiModel(description = "删除Connector")
|
||||
public class ConnectorDeleteDTO extends ClusterConnectorDTO {
|
||||
}
|
||||
@@ -1,20 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.dto.connect.task;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.connector.ConnectorActionDTO;
|
||||
import io.swagger.annotations.ApiModel;
|
||||
import io.swagger.annotations.ApiModelProperty;
|
||||
import lombok.Data;
|
||||
|
||||
import javax.validation.constraints.NotNull;
|
||||
|
||||
/**
|
||||
* @author zengqiao
|
||||
* @date 2022-10-17
|
||||
*/
|
||||
@Data
|
||||
@ApiModel(description = "操作Task")
|
||||
public class TaskActionDTO extends ConnectorActionDTO {
|
||||
@NotNull(message = "taskId不允许为NULL")
|
||||
@ApiModelProperty(value = "taskId", example = "123")
|
||||
private Long taskId;
|
||||
}
|
||||
@@ -1,22 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.connect;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.MetricDTO;
|
||||
import io.swagger.annotations.ApiModel;
|
||||
import io.swagger.annotations.ApiModelProperty;
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* @author didi
|
||||
*/
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
@ApiModel(description = "Connect集群指标查询信息")
|
||||
public class MetricsConnectClustersDTO extends MetricDTO {
|
||||
@ApiModelProperty("Connect集群ID")
|
||||
private List<Long> connectClusterIdList;
|
||||
}
|
||||
@@ -1,23 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.connect;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.connect.ClusterConnectorDTO;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.dto.metrices.MetricDTO;
|
||||
import io.swagger.annotations.ApiModel;
|
||||
import io.swagger.annotations.ApiModelProperty;
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* @author didi
|
||||
*/
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
@ApiModel(description = "Connector指标查询信息")
|
||||
public class MetricsConnectorsDTO extends MetricDTO {
|
||||
@ApiModelProperty("Connector列表")
|
||||
private List<ClusterConnectorDTO> connectorNameList;
|
||||
}
|
||||
@@ -3,7 +3,7 @@ package com.xiaojukeji.know.streaming.km.common.bean.entity;
|
||||
/**
|
||||
* @author didi
|
||||
*/
|
||||
public interface EntityIdInterface {
|
||||
public interface EntifyIdInterface {
|
||||
/**
|
||||
* 获取id
|
||||
* @return
|
||||
@@ -1,6 +1,6 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.cluster;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.EntityIdInterface;
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.EntifyIdInterface;
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
@@ -10,7 +10,7 @@ import java.util.Date;
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public class ClusterPhy implements Comparable<ClusterPhy>, EntityIdInterface {
|
||||
public class ClusterPhy implements Comparable<ClusterPhy>, EntifyIdInterface {
|
||||
/**
|
||||
* 主键
|
||||
*/
|
||||
|
||||
@@ -1,37 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.cluster;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
|
||||
/**
|
||||
* 集群状态信息
|
||||
* @author zengqiao
|
||||
* @date 22/02/24
|
||||
*/
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public class ClusterPhysHealthState {
|
||||
private Integer unknownCount;
|
||||
|
||||
private Integer goodCount;
|
||||
|
||||
private Integer mediumCount;
|
||||
|
||||
private Integer poorCount;
|
||||
|
||||
private Integer deadCount;
|
||||
|
||||
private Integer total;
|
||||
|
||||
public ClusterPhysHealthState(Integer total) {
|
||||
this.unknownCount = 0;
|
||||
this.goodCount = 0;
|
||||
this.mediumCount = 0;
|
||||
this.poorCount = 0;
|
||||
this.deadCount = 0;
|
||||
this.total = total;
|
||||
}
|
||||
}
|
||||
@@ -1,8 +1,8 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.config;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.Constant;
|
||||
import io.swagger.annotations.ApiModel;
|
||||
import io.swagger.annotations.ApiModelProperty;
|
||||
import lombok.Data;
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.util.Properties;
|
||||
@@ -11,6 +11,7 @@ import java.util.Properties;
|
||||
* @author zengqiao
|
||||
* @date 22/02/24
|
||||
*/
|
||||
@Data
|
||||
@ApiModel(description = "ZK配置")
|
||||
public class ZKConfig implements Serializable {
|
||||
@ApiModelProperty(value="ZK的jmx配置")
|
||||
@@ -20,51 +21,11 @@ public class ZKConfig implements Serializable {
|
||||
private Boolean openSecure = false;
|
||||
|
||||
@ApiModelProperty(value="ZK的Session超时时间", example = "15000")
|
||||
private Integer sessionTimeoutUnitMs = 15000;
|
||||
private Long sessionTimeoutUnitMs = 15000L;
|
||||
|
||||
@ApiModelProperty(value="ZK的Request超时时间", example = "5000")
|
||||
private Integer requestTimeoutUnitMs = 5000;
|
||||
private Long requestTimeoutUnitMs = 5000L;
|
||||
|
||||
@ApiModelProperty(value="ZK的Request超时时间")
|
||||
private Properties otherProps = new Properties();
|
||||
|
||||
public JmxConfig getJmxConfig() {
|
||||
return jmxConfig == null? new JmxConfig(): jmxConfig;
|
||||
}
|
||||
|
||||
public void setJmxConfig(JmxConfig jmxConfig) {
|
||||
this.jmxConfig = jmxConfig;
|
||||
}
|
||||
|
||||
public Boolean getOpenSecure() {
|
||||
return openSecure != null && openSecure;
|
||||
}
|
||||
|
||||
public void setOpenSecure(Boolean openSecure) {
|
||||
this.openSecure = openSecure;
|
||||
}
|
||||
|
||||
public Integer getSessionTimeoutUnitMs() {
|
||||
return sessionTimeoutUnitMs == null? Constant.DEFAULT_SESSION_TIMEOUT_UNIT_MS: sessionTimeoutUnitMs;
|
||||
}
|
||||
|
||||
public void setSessionTimeoutUnitMs(Integer sessionTimeoutUnitMs) {
|
||||
this.sessionTimeoutUnitMs = sessionTimeoutUnitMs;
|
||||
}
|
||||
|
||||
public Integer getRequestTimeoutUnitMs() {
|
||||
return requestTimeoutUnitMs == null? Constant.DEFAULT_REQUEST_TIMEOUT_UNIT_MS: requestTimeoutUnitMs;
|
||||
}
|
||||
|
||||
public void setRequestTimeoutUnitMs(Integer requestTimeoutUnitMs) {
|
||||
this.requestTimeoutUnitMs = requestTimeoutUnitMs;
|
||||
}
|
||||
|
||||
public Properties getOtherProps() {
|
||||
return otherProps == null? new Properties() : otherProps;
|
||||
}
|
||||
|
||||
public void setOtherProps(Properties otherProps) {
|
||||
this.otherProps = otherProps;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -13,4 +13,9 @@ public class BaseClusterHealthConfig extends BaseClusterConfigValue {
|
||||
* 健康检查名称
|
||||
*/
|
||||
protected HealthCheckNameEnum checkNameEnum;
|
||||
|
||||
/**
|
||||
* 权重
|
||||
*/
|
||||
protected Float weight;
|
||||
}
|
||||
|
||||
@@ -1,19 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.config.healthcheck;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @author wyb
|
||||
* @date 2022/10/26
|
||||
*/
|
||||
@Data
|
||||
public class HealthAmountRatioConfig extends BaseClusterHealthConfig {
|
||||
/**
|
||||
* 总数
|
||||
*/
|
||||
private Integer amount;
|
||||
/**
|
||||
* 比例
|
||||
*/
|
||||
private Double ratio;
|
||||
}
|
||||
@@ -1,5 +1,7 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.config.metric;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.constant.Constant;
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
|
||||
@@ -1,61 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.connect;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.bean.entity.EntityIdInterface;
|
||||
import lombok.Data;
|
||||
|
||||
import java.io.Serializable;
|
||||
|
||||
@Data
|
||||
public class ConnectCluster implements Serializable, Comparable<ConnectCluster>, EntityIdInterface {
|
||||
/**
|
||||
* 集群ID
|
||||
*/
|
||||
private Long id;
|
||||
|
||||
/**
|
||||
* 集群名字
|
||||
*/
|
||||
private String name;
|
||||
|
||||
/**
|
||||
* 集群使用的消费组
|
||||
*/
|
||||
private String groupName;
|
||||
|
||||
/**
|
||||
* 集群使用的消费组状态，也表示集群状态
|
||||
* @see com.xiaojukeji.know.streaming.km.common.enums.group.GroupStateEnum
|
||||
*/
|
||||
private Integer state;
|
||||
|
||||
/**
|
||||
* worker中显示的leader url信息
|
||||
*/
|
||||
private String memberLeaderUrl;
|
||||
|
||||
/**
|
||||
* 版本信息
|
||||
*/
|
||||
private String version;
|
||||
|
||||
/**
|
||||
* jmx配置
|
||||
* @see com.xiaojukeji.know.streaming.km.common.bean.entity.config.JmxConfig
|
||||
*/
|
||||
private String jmxProperties;
|
||||
|
||||
/**
|
||||
* Kafka集群ID
|
||||
*/
|
||||
private Long kafkaClusterPhyId;
|
||||
|
||||
/**
|
||||
* 集群地址
|
||||
*/
|
||||
private String clusterUrl;
|
||||
|
||||
@Override
|
||||
public int compareTo(ConnectCluster connectCluster) {
|
||||
return this.id.compareTo(connectCluster.getId());
|
||||
}
|
||||
}
|
||||
@@ -1,38 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.connect;
|
||||
|
||||
import com.xiaojukeji.know.streaming.km.common.enums.group.GroupStateEnum;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
import java.io.Serializable;
|
||||
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
public class ConnectClusterMetadata implements Serializable {
|
||||
/**
|
||||
* Kafka集群名字
|
||||
*/
|
||||
private Long kafkaClusterPhyId;
|
||||
|
||||
/**
|
||||
* 集群使用的消费组
|
||||
*/
|
||||
private String groupName;
|
||||
|
||||
/**
|
||||
* 集群使用的消费组状态，也表示集群状态
|
||||
*/
|
||||
private GroupStateEnum state;
|
||||
|
||||
/**
|
||||
* worker中显示的leader url信息
|
||||
*/
|
||||
private String memberLeaderUrl;
|
||||
|
||||
public ConnectClusterMetadata(Long kafkaClusterPhyId, String groupName, GroupStateEnum state, String memberLeaderUrl) {
|
||||
this.kafkaClusterPhyId = kafkaClusterPhyId;
|
||||
this.groupName = groupName;
|
||||
this.state = state;
|
||||
this.memberLeaderUrl = memberLeaderUrl;
|
||||
}
|
||||
}
|
||||
@@ -1,87 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.connect;
|
||||
|
||||
import com.didiglobal.logi.log.ILog;
|
||||
import com.didiglobal.logi.log.LogFactory;
|
||||
import com.xiaojukeji.know.streaming.km.common.utils.CommonUtils;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.net.URI;
|
||||
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
public class ConnectWorker implements Serializable {
|
||||
|
||||
protected static final ILog LOGGER = LogFactory.getLog(ConnectWorker.class);
|
||||
|
||||
/**
|
||||
* Kafka集群ID
|
||||
*/
|
||||
private Long kafkaClusterPhyId;
|
||||
|
||||
/**
|
||||
* 集群ID
|
||||
*/
|
||||
private Long connectClusterId;
|
||||
|
||||
/**
|
||||
* 成员ID
|
||||
*/
|
||||
private String memberId;
|
||||
|
||||
/**
|
||||
* 主机
|
||||
*/
|
||||
private String host;
|
||||
|
||||
/**
|
||||
* Jmx端口
|
||||
*/
|
||||
private Integer jmxPort;
|
||||
|
||||
/**
|
||||
* URL
|
||||
*/
|
||||
private String url;
|
||||
|
||||
/**
|
||||
* leader的URL
|
||||
*/
|
||||
private String leaderUrl;
|
||||
|
||||
/**
|
||||
* 1：是leader，0：不是leader
|
||||
*/
|
||||
private Integer leader;
|
||||
|
||||
/**
|
||||
* worker地址
|
||||
*/
|
||||
private String workerId;
|
||||
|
||||
public ConnectWorker(Long kafkaClusterPhyId,
|
||||
Long connectClusterId,
|
||||
String memberId,
|
||||
String host,
|
||||
Integer jmxPort,
|
||||
String url,
|
||||
String leaderUrl,
|
||||
Integer leader) {
|
||||
this.kafkaClusterPhyId = kafkaClusterPhyId;
|
||||
this.connectClusterId = connectClusterId;
|
||||
this.memberId = memberId;
|
||||
this.host = host;
|
||||
this.jmxPort = jmxPort;
|
||||
this.url = url;
|
||||
this.leaderUrl = leaderUrl;
|
||||
this.leader = leader;
|
||||
String workerId = CommonUtils.getWorkerId(url);
|
||||
if (workerId == null) {
|
||||
workerId = memberId;
|
||||
LOGGER.error("class=ConnectWorker||connectClusterId={}||memberId={}||url={}||msg=analysis url fail"
|
||||
, connectClusterId, memberId, url);
|
||||
}
|
||||
this.workerId = workerId;
|
||||
}
|
||||
}
|
||||
@@ -1,58 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.connect;
|
||||
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
import java.io.Serializable;
|
||||
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
public class WorkerConnector implements Serializable {
|
||||
/**
|
||||
* connect集群ID
|
||||
*/
|
||||
private Long connectClusterId;
|
||||
|
||||
/**
|
||||
* kafka集群ID
|
||||
*/
|
||||
private Long kafkaClusterPhyId;
|
||||
|
||||
/**
|
||||
* connector名称
|
||||
*/
|
||||
private String connectorName;
|
||||
|
||||
private String workerMemberId;
|
||||
|
||||
/**
|
||||
* 任务状态
|
||||
*/
|
||||
private String state;
|
||||
|
||||
/**
|
||||
* 任务ID
|
||||
*/
|
||||
private Integer taskId;
|
||||
|
||||
/**
|
||||
* worker信息
|
||||
*/
|
||||
private String workerId;
|
||||
|
||||
/**
|
||||
* 错误原因
|
||||
*/
|
||||
private String trace;
|
||||
|
||||
public WorkerConnector(Long kafkaClusterPhyId, Long connectClusterId, String connectorName, String workerMemberId, Integer taskId, String state, String workerId, String trace) {
|
||||
this.kafkaClusterPhyId = kafkaClusterPhyId;
|
||||
this.connectClusterId = connectClusterId;
|
||||
this.connectorName = connectorName;
|
||||
this.workerMemberId = workerMemberId;
|
||||
this.taskId = taskId;
|
||||
this.state = state;
|
||||
this.workerId = workerId;
|
||||
this.trace = trace;
|
||||
}
|
||||
}
|
||||
@@ -1,19 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.connect.config;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
import org.apache.kafka.connect.runtime.rest.entities.ConfigInfo;
|
||||
|
||||
|
||||
/**
|
||||
* @see ConfigInfo
|
||||
*/
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public class ConnectConfigInfo {
|
||||
private ConnectConfigKeyInfo definition;
|
||||
|
||||
private ConnectConfigValueInfo value;
|
||||
}
|
||||
@@ -1,71 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.connect.config;
|
||||
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
import org.apache.kafka.connect.runtime.rest.entities.ConfigInfo;
|
||||
import org.apache.kafka.connect.runtime.rest.entities.ConfigInfos;
|
||||
|
||||
import java.util.*;
|
||||
|
||||
import static com.xiaojukeji.know.streaming.km.common.constant.Constant.CONNECTOR_CONFIG_ACTION_RELOAD_NAME;
|
||||
import static com.xiaojukeji.know.streaming.km.common.constant.Constant.CONNECTOR_CONFIG_ERRORS_TOLERANCE_NAME;
|
||||
|
||||
/**
|
||||
* @see ConfigInfos
|
||||
*/
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public class ConnectConfigInfos {
|
||||
|
||||
private static final Map<String, List<String>> recommendValuesMap = new HashMap<>();
|
||||
|
||||
static {
|
||||
recommendValuesMap.put(CONNECTOR_CONFIG_ACTION_RELOAD_NAME, Arrays.asList("none", "restart"));
|
||||
recommendValuesMap.put(CONNECTOR_CONFIG_ERRORS_TOLERANCE_NAME, Arrays.asList("none", "all"));
|
||||
}
|
||||
private String name;
|
||||
|
||||
private int errorCount;
|
||||
|
||||
private List<String> groups;
|
||||
|
||||
private List<ConnectConfigInfo> configs;
|
||||
|
||||
public ConnectConfigInfos(ConfigInfos configInfos) {
|
||||
this.name = configInfos.name();
|
||||
this.errorCount = configInfos.errorCount();
|
||||
this.groups = configInfos.groups();
|
||||
|
||||
this.configs = new ArrayList<>();
|
||||
for (ConfigInfo configInfo: configInfos.values()) {
|
||||
ConnectConfigKeyInfo definition = new ConnectConfigKeyInfo();
|
||||
definition.setName(configInfo.configKey().name());
|
||||
definition.setType(configInfo.configKey().type());
|
||||
definition.setRequired(configInfo.configKey().required());
|
||||
definition.setDefaultValue(configInfo.configKey().defaultValue());
|
||||
definition.setImportance(configInfo.configKey().importance());
|
||||
definition.setDocumentation(configInfo.configKey().documentation());
|
||||
definition.setGroup(configInfo.configKey().group());
|
||||
definition.setOrderInGroup(configInfo.configKey().orderInGroup());
|
||||
definition.setWidth(configInfo.configKey().width());
|
||||
definition.setDisplayName(configInfo.configKey().displayName());
|
||||
definition.setDependents(configInfo.configKey().dependents());
|
||||
|
||||
ConnectConfigValueInfo value = new ConnectConfigValueInfo();
|
||||
value.setName(configInfo.configValue().name());
|
||||
value.setValue(configInfo.configValue().value());
|
||||
value.setRecommendedValues(recommendValuesMap.getOrDefault(configInfo.configValue().name(), configInfo.configValue().recommendedValues()));
|
||||
value.setErrors(configInfo.configValue().errors());
|
||||
value.setVisible(configInfo.configValue().visible());
|
||||
|
||||
ConnectConfigInfo connectConfigInfo = new ConnectConfigInfo();
|
||||
connectConfigInfo.setDefinition(definition);
|
||||
connectConfigInfo.setValue(value);
|
||||
|
||||
this.configs.add(connectConfigInfo);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,38 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.connect.config;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
import org.apache.kafka.connect.runtime.rest.entities.ConfigKeyInfo;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* @see ConfigKeyInfo
|
||||
*/
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public class ConnectConfigKeyInfo {
|
||||
private String name;
|
||||
|
||||
private String type;
|
||||
|
||||
private boolean required;
|
||||
|
||||
private String defaultValue;
|
||||
|
||||
private String importance;
|
||||
|
||||
private String documentation;
|
||||
|
||||
private String group;
|
||||
|
||||
private int orderInGroup;
|
||||
|
||||
private String width;
|
||||
|
||||
private String displayName;
|
||||
|
||||
private List<String> dependents;
|
||||
}
|
||||
@@ -1,27 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.connect.config;
|
||||
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
import org.apache.kafka.connect.runtime.rest.entities.ConfigValueInfo;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* @see ConfigValueInfo
|
||||
*/
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public class ConnectConfigValueInfo {
|
||||
private String name;
|
||||
|
||||
private String value;
|
||||
|
||||
private List<String> recommendedValues;
|
||||
|
||||
private List<String> errors;
|
||||
|
||||
private boolean visible;
|
||||
}
|
||||
@@ -1,20 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.connect.connector;
|
||||
|
||||
import com.alibaba.fastjson.annotation.JSONField;
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
import lombok.Data;
|
||||
import org.apache.kafka.connect.runtime.rest.entities.ConnectorStateInfo;
|
||||
|
||||
/**
|
||||
* @see ConnectorStateInfo.AbstractState
|
||||
*/
|
||||
@Data
|
||||
public abstract class KSAbstractConnectState {
|
||||
private String state;
|
||||
|
||||
private String trace;
|
||||
|
||||
@JSONField(name="worker_id")
|
||||
@JsonProperty("worker_id")
|
||||
private String workerId;
|
||||
}
|
||||
@@ -1,48 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.connect.connector;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
import java.io.Serializable;
|
||||
|
||||
@Data
|
||||
public class KSConnector implements Serializable {
|
||||
/**
|
||||
* Kafka集群ID
|
||||
*/
|
||||
private Long kafkaClusterPhyId;
|
||||
|
||||
/**
|
||||
* connect集群ID
|
||||
*/
|
||||
private Long connectClusterId;
|
||||
|
||||
/**
|
||||
* connector名称
|
||||
*/
|
||||
private String connectorName;
|
||||
|
||||
/**
|
||||
* connector类名
|
||||
*/
|
||||
private String connectorClassName;
|
||||
|
||||
/**
|
||||
* connector类型
|
||||
*/
|
||||
private String connectorType;
|
||||
|
||||
/**
|
||||
* 访问过的Topic列表
|
||||
*/
|
||||
private String topics;
|
||||
|
||||
/**
|
||||
* task数
|
||||
*/
|
||||
private Integer taskCount;
|
||||
|
||||
/**
|
||||
* 状态
|
||||
*/
|
||||
private String state;
|
||||
}
|
||||
@@ -1,26 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.connect.connector;
|
||||
|
||||
import lombok.Data;
|
||||
import org.apache.kafka.connect.runtime.rest.entities.ConnectorType;
|
||||
import org.apache.kafka.connect.util.ConnectorTaskId;
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* copy from:
|
||||
* @see org.apache.kafka.connect.runtime.rest.entities.ConnectorInfo
|
||||
*/
|
||||
@Data
|
||||
public class KSConnectorInfo implements Serializable {
|
||||
private Long connectClusterId;
|
||||
|
||||
private String name;
|
||||
|
||||
private Map<String, String> config;
|
||||
|
||||
private List<ConnectorTaskId> tasks;
|
||||
|
||||
private ConnectorType type;
|
||||
}
|
||||
@@ -1,11 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.connect.connector;
|
||||
|
||||
import lombok.Data;
|
||||
import org.apache.kafka.connect.runtime.rest.entities.ConnectorStateInfo;
|
||||
|
||||
/**
|
||||
* @see ConnectorStateInfo.ConnectorState
|
||||
*/
|
||||
@Data
|
||||
public class KSConnectorState extends KSAbstractConnectState {
|
||||
}
|
||||
@@ -1,21 +0,0 @@
|
||||
package com.xiaojukeji.know.streaming.km.common.bean.entity.connect.connector;
|
||||
|
||||
import lombok.Data;
|
||||
import org.apache.kafka.connect.runtime.rest.entities.ConnectorStateInfo;
|
||||
import org.apache.kafka.connect.runtime.rest.entities.ConnectorType;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* @see ConnectorStateInfo
|
||||
*/
|
||||
@Data
|
||||
public class KSConnectorStateInfo {
|
||||
private String name;
|
||||
|
||||
private KSConnectorState connector;
|
||||
|
||||
private List<KSTaskState> tasks;
|
||||
|
||||
private ConnectorType type;
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user