One near-failure story to keep myself writing. I almost reported an issue, right up to the point where bisecting doesn't reproduce it reliably.
I run both kafka-minion [v1, known under cloudhut org](https://github.com/redpanda-data/kminion/blob/v1.0.2/options/options.go#L63) and kminion [v2, now under redpanda-data org](https://github.com/redpanda-data/kminion/blob/v2.2.0/docs/reference-config.yaml#L13) in parallel, in order to gauge how their metrics compare. (Truth be told, i wrote some suboptimal configs for kafka-minion, and the dashboards are too tied up with such config.) They use the same configuration via a systemd `EnvironmentFile`. That's the only method supported by kafka-minion whose `Options` are deserialized by struct definion:
> string `envconfig:"KAFKA_TLS_CA_FILE_PATH"`
kminion can load config from some `koanf` providers: env and/or yaml. There's a transformation step to collapse yaml maps into a flat environment variable name:
> kafka.tls.caFilepath => `KAFKA_TLS_CAFILEPATH`
Due to some recent issue, I wanted to build kminion from git repo. However, moving beyond https://github.com/redpanda-data/kminion/pull/147 gave a weird error:
```json
  "caller": "kminion/main.go:43",
  "msg": "failed to parse config",
  "error": "1 error(s) decoding:\n\n* 'kafka.tls.ca[0]' expected type 'uint8', got unconvertible type 'map[string]interface {}', value: 'map[file:map[path:kafka_chain.pem]]'",
  "stacktrace": "main.main\n\t/home/me/github/kminion/main.go:43\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:250"
```
`map[file:map[path:kafka_chain.pem]]` okay, I see my CA file here (actually the content is ISRG root CA). But what's this `'kafka.tls.ca[0]' expected type 'uint8'`? Why is something plugging a map of into an integer? At least I have a clean commit to write a bisect demo.
So I copied over the `EnvironmentFile`, removed anything about `SASL` authentication, exporter, set `0:9092` as `bootstrap.servers` for a quick failure once the app passes config parsing. And let `git bisect run` rip!
It pointed somewhere around the merge commit, as i expected. But then i had doubt, and set kafka-minion's envvar in a shell:
```sh
env KAFKA_TLS_CAFILEPATH=kafka_chain.pem kafka-minion
...snip...Connection refused
```
eh, it got right up to trying connection? what happened to my bug?
As it happened, the envvar intended for `kafka-minion` (`KAFKA_TLS_CA_FILE_PATH`) matches `koanf` auto-hydration(?), and it tried to construct this structure:
```yaml
kafka:
  tls:
    ca:
      file:
        path: "kafka_chain.pem"
```
because the new feature now accepts this structure:
```yaml
kafka:
  tls:
    ca: MIIF....base64 DER data....
```
Thus, the `expected` first element should have been a byte from the certificate.
With the new-issue tab firmly closed, I added to kminion's systemd unit file:
```conf
[Service]
UnsetEnvironment=KAFKA_TLS_CA_FILE_PATH
```
As usual, I succeeded by nearly failing.

ln -svf $PWD/gitconfig ~/.gitconfig
ln -svf $PWD/pijul_config.toml ~/.config/pijul/config.toml

   '(rustic markdown-mode markdownfmt mermaid-mode tree-sitter-langs json-mode eglot rust-mode go-mode jinja2-mode dockerfile-mode company-shell groovy-mode rainbow-delimiters yaml-mode modus-themes magit-delta use-package))

   '(lsp-mode rustic markdown-mode markdownfmt mermaid-mode tree-sitter-langs json-mode eglot rust-mode go-mode jinja2-mode dockerfile-mode company-shell groovy-mode rainbow-delimiters yaml-mode modus-themes magit-delta use-package))

  (setq rustic-format-on-save t))

  (setq rustic-format-on-save nil))

[user]
name = hdhoang
email = code@hdhoang.space

# verify with `readelf -p .comment`

# verify with `readelf -p .comment`

**/*~
**/*.bk
.pijul
.git
.DS_Store