Option to merge the JVM truststore with user-supplied truststore

[thelabdude]

Fixes #390 ~ by allowing the JVM cacerts to get merged in with the user-supplied truststore (Let's Encrypt's CA is in the cacerts for modern JVM)

Users can now configure the TLS options to merge the JVM's truststore with the truststore for their certs using:

spec:
  ...
  solrTLS:
    ...
    trustStoreSecret:
      name: dev-selfsigned-cert-tls
      key: truststore.p12
    mergeJavaTrustStore: "$JAVA_HOME/lib/security/cacerts"

The path given in mergeJavaTrustStore option must exist on the Solr docker image! Thus, if users customize their Solr image, this path may be different.

Behind the scenes, this creates an additional initContainer that merges the two truststores together and then points the env var to the "merged" truststore. The actual merging is done with keytool.

For server TLS:

- name: SOLR_SSL_TRUST_STORE
  value: /var/solr/tls-merged/truststore.p12

By pointing SOLR_SSL_TRUST_STORE env var at the merged truststore, we're ensured that all the other initContainers and liveness probes continue to work (as they just use the env var to resolve this path).

Added a few simple unit tests and tested manually in my cluster.

For Prom exporter, the config would be:

spec:
  solrReference:
    ...
    solrTLS:
      ...
      mergeJavaTrustStore: "$JAVA_HOME/lib/security/cacerts"

Which results in the exporter container getting configed with env var:

- name: SOLR_SSL_CLIENT_TRUST_STORE
  value: /var/solr/tls-merged/truststore.p12

[HoustonPutman]

Would you be able to add something to the docs about this as well?

Also a changelog entry would be much appreciated 🙂

[HoustonPutman]

Can we add the // +optional tag for both of these new options? Just for consistency sake

[HoustonPutman]

Should this be a secret reference? How bad is it to have this in plain text?

[thelabdude]

seemed like overkill to me since it's the truststore pass for the JVM which is most likely "changeit" and isn't used by Solr ... that said, we can pull from a secret too ... doubt many would ever use this option.

[HoustonPutman]

ahhh ok, didn't understand it was unusual to change it. Sounds good

[HoustonPutman]

Could we possibly use the default initContainer resources?

You can see how it was done in this PR: https://github.com/apache/solr-operator/pull/451/files#diff-653faf42eabedf3285e433f247c993282f035ee70781d151f8c8d68fee2621a3R618-R649

[HoustonPutman]

Does this work with mountedTLSDir?

[thelabdude]

Typically, mountedTLSDir will have a csi driver volume and corresponding mount on the mainContainer, which would get used by the merge initContainer, so the init container would get the truststore file. However, that might cause double creation of the cert for each pod, once for the initContainer and once for the main container, so this would likely put a lot of pressure on the Cert issuer. So probably safer to say this feature is not supported with mountedTLSDir option for now.

[thelabdude]

No, nvm all that volumesAndMounts doesn't include vols & mounts for the mountedTLSDir option. So this option is not going to work with mountedTLSDir.

So maybe we just punt on this feature for 0.6 and solve it using an init-db script instead? There's already some code in place for mounting a script into the init-db.

[thelabdude]

I think we should hold off on this solution for 0.6 as now I'm thinking the better approach would be to use the init-db script approach built into the Solr image. The problem with the current solution in this PR is it doesn't work with the mountedTLSDir option. Originally I was thinking the volumes and mounts would be available to the merge initContainer but that's not the case. I think the init-db approach will actually be fairly easy to implement except for the Solr exporter. Unfortunately, I won't have time to work on this for a bit and I don't want to hold up 0.6

[HoustonPutman]

Sounds good Tim. Thanks for putting in the effort on this though!