diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md new file mode 100644 index 00000000..afbe0a05 --- /dev/null +++ b/.github/ISSUE_TEMPLATE.md @@ -0,0 +1 @@ +## Issue 模板 \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 00000000..30e38e30 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,5 @@ +blank_issues_enabled: true +contact_links: + - name: KnowStreaming官网 + url: https://knowstreaming.com/ + about: KnowStreaming website diff --git a/.github/ISSUE_TEMPLATE/detail_optimizing.md b/.github/ISSUE_TEMPLATE/detail_optimizing.md new file mode 100644 index 00000000..a531ac8a --- /dev/null +++ b/.github/ISSUE_TEMPLATE/detail_optimizing.md @@ -0,0 +1,22 @@ +--- +name: 优化建议 +about: 相关功能优化建议 +title: '' +labels: Optimization Suggestions +assignees: '' + +--- + +- [ ] 我已经在 [issues](https://github.com/didi/KnowStreaming/issues) 搜索过相关问题了,并没有重复的。 + +### 环境信息 + +* KnowStreaming version : xxx +* Operating System version : xxx +* Java version : xxx + +### 需要优化的功能点 + + +### 建议如何优化 + diff --git a/.github/ISSUE_TEMPLATE/discussion.md b/.github/ISSUE_TEMPLATE/discussion.md index 7c658e51..bf2b8359 100644 --- a/.github/ISSUE_TEMPLATE/discussion.md +++ b/.github/ISSUE_TEMPLATE/discussion.md @@ -1,9 +1,10 @@ --- -name: 讨论 +name: 讨论/discussion about: 开启一个关于KnowStreaming的讨论 title: '' labels: discussion assignees: '' + --- ## 讨论主题 diff --git a/.github/ISSUE_TEMPLATE/question.md b/.github/ISSUE_TEMPLATE/question.md index 784dfbec..e9bf5c05 100644 --- a/.github/ISSUE_TEMPLATE/question.md +++ b/.github/ISSUE_TEMPLATE/question.md @@ -1,6 +1,6 @@ --- -name: 问题/Question -about: 问KnowStreaming相关问题, 如果没有得到及时回复可以加V: shiyanzu001 , 进交流群 +name: 提个问题 +about: 问KnowStreaming相关问题 title: '' labels: question assignees: '' diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 00000000..a70c8889 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,74 @@ + +# Contributor Covenant Code of Conduct + +## Our Pledge + +In the interest of fostering an open and welcoming environment, we as +contributors and maintainers pledge to making participation in our project and +our community a harassment-free experience for everyone, regardless of age, body +size, disability, ethnicity, gender identity and expression, level of experience, +education, socio-economic status, nationality, personal appearance, race, +religion, or sexual identity and orientation. + +## Our Standards + +Examples of behavior that contributes to creating a positive environment +include: + +* Using welcoming and inclusive language +* Being respectful of differing viewpoints and experiences +* Gracefully accepting constructive criticism +* Focusing on what is best for the community +* Showing empathy towards other community members + +Examples of unacceptable behavior by participants include: + +* The use of sexualized language or imagery and unwelcome sexual attention or + advances +* Trolling, insulting/derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or electronic + address, without explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Our Responsibilities + +Project maintainers are responsible for clarifying the standards of acceptable +behavior and are expected to take appropriate and fair corrective action in +response to any instances of unacceptable behavior. + +Project maintainers have the right and responsibility to remove, edit, or +reject comments, commits, code, wiki edits, issues, and other contributions +that are not aligned to this Code of Conduct, or to ban temporarily or +permanently any contributor for other behaviors that they deem inappropriate, +threatening, offensive, or harmful. + +## Scope + +This Code of Conduct applies both within project spaces and in public spaces +when an individual is representing the project or its community. Examples of +representing a project or community include using an official project e-mail +address, posting via an official social media account, or acting as an appointed +representative at an online or offline event. Representation of a project may be +further defined and clarified by project maintainers. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported by contacting the project team at shirenchuang@didiglobal.com . All +complaints will be reviewed and investigated and will result in a response that +is deemed necessary and appropriate to the circumstances. The project team is +obligated to maintain confidentiality with regard to the reporter of an incident. +Further details of specific enforcement policies may be posted separately. + +Project maintainers who do not follow or enforce the Code of Conduct in good +faith may face temporary or permanent repercussions as determined by other +members of the project's leadership. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, +available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html + +[homepage]: https://www.contributor-covenant.org diff --git a/README.md b/README.md index 1ae6ca59..cccf3960 100644 --- a/README.md +++ b/README.md @@ -134,6 +134,11 @@ PS: 提问请尽量把问题一次性描述清楚，并告知环境信息情况 微信加群：添加`mike_zhangliang`、`PenceXie`的微信号备注KnowStreaming加群。
+ +加群之前有劳点一下 star，一个小小的 star 是对KnowStreaming作者们努力建设社区的动力。 + +感谢感谢！！！ + wx ## Star History diff --git a/docs/dev_guide/assets/support_kerberos_zk/need_modify_code.png b/docs/dev_guide/assets/support_kerberos_zk/need_modify_code.png new file mode 100644 index 00000000..c51c1c00 Binary files /dev/null and b/docs/dev_guide/assets/support_kerberos_zk/need_modify_code.png differ diff --git a/docs/dev_guide/assets/support_kerberos_zk/success_1.png b/docs/dev_guide/assets/support_kerberos_zk/success_1.png new file mode 100644 index 00000000..f15ed55e Binary files /dev/null and b/docs/dev_guide/assets/support_kerberos_zk/success_1.png differ diff --git a/docs/dev_guide/assets/support_kerberos_zk/success_2.png b/docs/dev_guide/assets/support_kerberos_zk/success_2.png new file mode 100644 index 00000000..f15ed55e Binary files /dev/null and b/docs/dev_guide/assets/support_kerberos_zk/success_2.png differ diff --git a/docs/dev_guide/assets/support_kerberos_zk/watch_user_acl.png b/docs/dev_guide/assets/support_kerberos_zk/watch_user_acl.png new file mode 100644 index 00000000..b076316b Binary files /dev/null and b/docs/dev_guide/assets/support_kerberos_zk/watch_user_acl.png differ diff --git a/docs/dev_guide/支持Kerberos认证的ZK.md b/docs/dev_guide/支持Kerberos认证的ZK.md new file mode 100644 index 00000000..116643ba --- /dev/null +++ b/docs/dev_guide/支持Kerberos认证的ZK.md @@ -0,0 +1,69 @@ + +## 支持Kerberos认证的ZK + + +### 1、修改 KnowStreaming 代码 + +代码位置：`src/main/java/com/xiaojukeji/know/streaming/km/persistence/kafka/KafkaAdminZKClient.java` + +将 `createZKClient` 的 `135行 的 false 改为 true +![need_modify_code.png](assets/support_kerberos_zk/need_modify_code.png) + + +修改完后重新进行打包编译，打包编译见：[打包编译](https://github.com/didi/KnowStreaming/blob/master/docs/install_guide/%E6%BA%90%E7%A0%81%E7%BC%96%E8%AF%91%E6%89%93%E5%8C%85%E6%89%8B%E5%86%8C.md +) + + + +### 2、查看用户在ZK的ACL + +假设我们使用的用户是 `kafka` 这个用户。 + +- 1、查看 server.properties 的配置的 zookeeper.connect 的地址； +- 2、使用 `zkCli.sh -serve zookeeper.connect的地址` 登录到ZK页面； +- 3、ZK页面上，执行命令 `getAcl /kafka` 查看 `kafka` 用户的权限； + +此时，我们可以看到如下信息： +![watch_user_acl.png](assets/support_kerberos_zk/watch_user_acl.png) + +`kafka` 用户需要的权限是 `cdrwa`。如果用户没有 `cdrwa` 权限的话，需要创建用户并授权，授权命令为：`setAcl` + + +### 3、创建Kerberos的keytab并修改 KnowStreaming 主机 + +- 1、在 Kerberos 的域中创建 `kafka/_HOST` 的 `keytab`，并导出。例如：`kafka/dbs-kafka-test-8-53`； +- 2、导出 keytab 后上传到安装 KS 的机器的 `/etc/keytab` 下； +- 3、在 KS 机器上，执行 `kinit -kt zookeepe.keytab kafka/dbs-kafka-test-8-53` 看是否能进行 `Kerberos` 登录； +- 4、可以登录后，配置 `/opt/zookeeper.jaas` 文件，例子如下： +```sql +Client { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + storeKey=false + serviceName="zookeeper" + keyTab="/etc/keytab/zookeeper.keytab" + principal="kafka/dbs-kafka-test-8-53@XXX.XXX.XXX"; +}; +``` +- 5、需要配置 `KDC-Server` 对 `KnowStreaming` 的机器开通防火墙，并在KS的机器 `/etc/host/` 配置 `kdc-server` 的 `hostname`。并将 `krb5.conf` 导入到 `/etc` 下； + + +### 4、修改 KnowStreaming 的配置 + +- 1、在 `/usr/local/KnowStreaming/KnowStreaming/bin/startup.sh` 中的47行的JAVA_OPT中追加如下设置 +```bash +-Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/opt/zookeeper.jaas +``` + +- 2、重启KS集群后再 start.out 中看到如下信息，则证明Kerberos配置成功； + +![success_1.png](assets/support_kerberos_zk/success_1.png) + +![success_2.png](assets/support_kerberos_zk/success_2.png) + + +### 5、补充说明 + +- 1、多Kafka集群如果用的是一样的Kerberos域的话，只需在每个`ZK`中给`kafka`用户配置`crdwa`权限即可，这样集群初始化的时候`zkclient`是都可以认证； +- 2、当前需要修改代码重新打包才可以支持，后续考虑通过页面支持Kerberos认证的ZK接入； +- 3、多个Kerberos域暂时未适配； \ No newline at end of file diff --git a/km-persistence/src/main/java/com/xiaojukeji/know/streaming/km/persistence/kafka/KafkaAdminZKClient.java b/km-persistence/src/main/java/com/xiaojukeji/know/streaming/km/persistence/kafka/KafkaAdminZKClient.java index c8849ee7..e6275a60 100644 --- a/km-persistence/src/main/java/com/xiaojukeji/know/streaming/km/persistence/kafka/KafkaAdminZKClient.java +++ b/km-persistence/src/main/java/com/xiaojukeji/know/streaming/km/persistence/kafka/KafkaAdminZKClient.java @@ -133,6 +133,8 @@ public class KafkaAdminZKClient extends AbstractClusterLoadedChangedHandler impl kafkaZkClient = KafkaZkClient.apply( zookeeperAddress, false, +// 添加支持zk的Kerberos认证 +// true, Constant.DEFAULT_SESSION_TIMEOUT_UNIT_MS, Constant.DEFAULT_SESSION_TIMEOUT_UNIT_MS, 5,