Enabling Ceph in Proxmox is a fantastic way to provide scalable and redundant storage for your VMs.
Enabling the Amazon S3 interface for Ceph opens your storage up to third party applications that require a standard storage interface.
One the main concerns we had about doing this was that the hypervisors would be directly exposed via S3. And although S3 is a well established protocol, I don't believe it is good practice to expose hypervisors in any way. With that said, I decided to create a VM, add the VM to the Proxmox cluster and then expose S3 on the VM.
Without too much waffling, let's get started :)
Assume we have three nodes (node1, node2, node3) in a Proxmox cluster.
Start by creating the keyring on node1:
Now generate the keys and add them to the keyring created above:
Next, add the capabilities to each of the keys:
Now add the keys to the cluster:
Copy the keyring in to the Proxmox ClusterFS:
Edit /etc/ceph/ceph.conf and paste the text below in this file. Be sure to change s3.yourdomain.com to the domain you want to use to access the S3 interface.
Next, login to each of the nodes and install the radosgw package:
Next, start the gateway on each of the nodes:
Now enable the following on node1:
Create the admin user:
By default radosgw creates a default pool which might not be desired. In this case you can set the default pool which is useful if you have dedicated SSD and HDD pools. Assuming the pool you want S3 to use is called hdd_pool with a placement group called hddgroup and index pool called hddgroupindex you can run the following commands to set the default pool: