Original source: https://input.scs.community/how-to-s3#; written by Friedrich Zahn.

Swift+S3 on Rook¶

The official "guide" contains various inaccuracies and there are various caveats when deploying this setup on Yaook.

The Ceph Operator will not always immediately propagate config changes. Make sure to delete the rgw ReplicaSet often!

Preparations¶

Keystone user credentials¶

Copy the keystone-admin Secret from the yaook namespace to the rook-ceph namespace.

You MUST add OS_IDENTITY_API_VERSION: Mw== (decoded value: 3), otherwise the rgw will fall back to Keystone API v2.0, which is deprecated since ca. forever.

Alternatively, you can create a KeystoneUser, but that is discouraged by the docs.

CA bundle¶

Get the keystone-ca-certificates-... ConfigMap. Extract the value (loooong cert chain) for ca-bundle.crt

Create (not apply! Annotation may exceed 256 kiB) a Secret keystone-ca-certificates in the rook-ceph namespace with that chain (base64 encoded!) under the key cabundle

TLS Certificates for endpoints¶

We need a ClusterIssuer and Certificate:

 1apiVersion: cert-manager.io/v1
 2kind: ClusterIssuer
 3metadata:
 4  name: ca-issuer
 5spec:
 6  ca:
 7    secretName: root-ca
 8
 9
10apiVersion: cert-manager.io/v1
11kind: Certificate
12metadata:
13  name: ceph-rgw-objectstorage
14  namespace: rook-ceph
15spec:
16  commonName: ceph-rgw-objectstorage
17  dnsNames:
18  - rook-ceph-rgw-objectstorage.rook-ceph.svc
19  duration: 720h
20  issuerRef:
21    name: ca-issuer
22    kind: ClusterIssuer
23  renewBefore: 168h
24  revisionHistoryLimit: 3
25  secretName: ceph-rgw-objectstorage-certificate
26  subject:
27    organizations:
28    - yaook

This certificate is not recognized by the yaookctl openstack shell, thus all interaction with the endpoint has to be via http or with TLS verification disabled.

Roll-out¶

 1apiVersion: ceph.rook.io/v1
 2kind: CephObjectStore
 3metadata:
 4  name: objectstorage
 5  namespace: rook-ceph
 6spec:
 7  metadataPool:
 8    failureDomain: host
 9    replicated:
10      size: 1
11  dataPool:
12    failureDomain: host
13    replicated:
14      size: 1
15  auth:
16    keystone:
17      acceptedRoles:
18        - admin
19        - member
20        - service
21      implicitTenants: "true"
22      revocationInterval: 1200
23      serviceUserSecretName: keystone-admin
24      tokenCacheSize: 1000
25      url: https://keystone.yaook.svc:5000
26  protocols:
27    swift:
28      accountInUrl: true
29      urlPrefix: swift
30    s3:
31      authUseKeystone: true
32    # note that s3 is enabled by default if protocols.s3.enabled is not explicitly set to false
33  preservePoolsOnDelete: true
34  gateway:
35    caBundleRef: keystone-ca-certificates
36    sslCertificateRef: ceph-rgw-objectstorage-certificate
37    port: 80
38    securePort: 443
39    instances: 1

OpenStack integration¶

::: warn openstackclient caches the catalog. Do not use interactive mode to check it.

:::

Using openstackclient to add the service and endpoints (guide has wrong URLs):

1openstack service create --name swift object-store
2
3openstack endpoint create swift internal "https://rook-ceph-rgw-objectstorage.rook-ceph.svc/swift/v1/AUTH_%(tenant_id)s" --region YaookRegion
4
5openstack endpoint create swift admin "https://rook-ceph-rgw-objectstorage.rook-ceph.svc/swift/v1/AUTH_%(tenant_id)s" --region YaookRegion

Alternatively you can create a KeystoneEndpoint, although that is discouraged by the docs:

 1apiVersion: yaook.cloud/v1
 2kind: KeystoneEndpoint
 3metadata:
 4  name: ceph-rgw-objectstorage-endpoint
 5  namespace: yaook
 6spec:
 7  description: OpenStack Object Storage
 8  endpoints:
 9    admin: https://rook-ceph-rgw-objectstorage.rook-ceph.svc/swift/v1/AUTH_%(tenant_id)s
10    internal: https://rook-ceph-rgw-objectstorage.rook-ceph.svc/swift/v1/AUTH_%(tenant_id)s
11    public: https://rook-ceph-rgw-objectstorage.rook-ceph.svc/swift/v1/AUTH_%(tenant_id)s
12  keystoneRef:
13    kind: KeystoneDeployment
14    name: keystone
15  region:
16    name: YaookRegion
17  servicename: swift
18  servicetype: object-store

Testing¶

E.g. on a yaookctl openstack shell :

1openstack --insecure container create foobar
2openstack --insecure container list

1openstack ec2 credentials create
2pip install boto3
3python
4import boto3
5s3 = boto3.client('s3', aws_access_key_id="<access>", aws_secret_access_key="<secret>", endpoint_url="https://rook-ceph-rgw-objectstorage.rook-ceph.svc", verify=False)
6s3.create_bucket(Bucket="trollololol")
7s3.list_buckets()

Findings¶

the Rook/Ceph guide on swift emulation does not work as-is, there are wrong URLs, missing components, too little explanation
it is way too easy to mess this up, and debugging is super painful due to the many layers of services and logs that are very verbose, but often contain very little actual information
scs-check in the end still fails due to self-signed certificates