Set up image service with Thumbor and AWS S3 | by Quân Huỳnh

We are going to learn how to install Thumbor on a Kubernetes environment and use AWS S3 as image storage for Thumbor in this post.

Thumbor is a smart imaging service. It enables on-demand crop, resizing, and flipping of images.

It features a VERY smart detection of important points in the image for better cropping and resizing, using state-of-the-art face and feature detection algorithms.

Using thumbor is very easy. For example, we use Thumbor to access an image with a width of 300 and a length of 200.

http://thumbor-server/K97LekICOXT9MbO3X1u8BBkrjbu5/300x200/smart/example.jpg

Then if we need to access the image with a width of 350 and a length of 300, just change the value from 300×200 to 350×300.

Now, I will guide you on how to set up Thumbor on Kubernetes.

First, you create a file named thumbor-deployment.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
name: thumbor-server
spec:
replicas: 1
selector:
matchLabels:
app: thumbor
template:
metadata:
labels:
app: thumbor
spec:
containers:
- name: thumbor
image: minimalcompact/thumbor:6.7.5
command: ["thumbor"]
args: ["-p", "80", "-c", "/conf/thumbor.conf"]
ports:
- containerPort: 80
volumeMounts:
- name: thumbor-conf
mountPath: /conf
volumes:
- name: thumbor-conf
configMap:
name: thumbor

We will use Configmap to store the configuration of the thumbor server and mount it into Pod at a path /confand using two properties commandand argsto specify the thumbor runs on port 80 and use a file /conf/thumbor.conf as the configuration file.

Next, you create a file named thumbor.cm.yaml.

apiVersion: v1
kind: ConfigMap
metadata:
name: thumbor
data:
thumbor.conf: |
UPLOAD_ENABLED = True

In a thumbor.cm.yaml there will be an attribute UPLOAD_ENABLED = True for the thumb to enable upload file. Full configuration at this link https://thumbor.readthedocs.io/en/latest/configuration.html.

Next, you create a Service and Ingress to expose traffic of the thumbor server for an outside client. Creating a file named thumbor.svc.yaml.

apiVersion: v1
kind: Service
metadata:
name: thumbor
spec:
selector:
app: thumbor
ports:
- port: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: thumbor
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/healthcheck-path: /healthcheck
spec:
rules:
- host: thumbor.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: thumbor
port:
number: 80

In this article, because I use AWS Kubernetes Service, I use aws-load-balancer-controller for Ingress, you can see how to install it at this link https://aws.amazon.com/premiumsupport/knowledge-center/eks-alb-ingress-controller-fargate. As for the on-premises environment, you should use NGINX Ingress.

And if you want to test on a local environment, we create a NodePort Service.

apiVersion: v1
kind: Service
metadata:
name: thumbor
spec:
type: NodePort
selector:
app: thumbor
ports:
- port: 80
nodePort: 30000

Now, we create resources on Kubernetes by running the following commands.

kubectl apply -f thumbor.cm.yaml
kubectl apply -f thumbor-deployment.yaml
kubectl apply -f thumbor.svc.yaml

Let’s check it, we upload a file to the thumbor server.

curl -i -H "Content-Type: image/jpeg" -H "Slug: photo.jpg" -XPOST http://thumbor.example.com/image --data-binary "@example.jpg"

Result.

HTTP/1.1 201 Created
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Location: /image/05b2eda857314e559630c6f3334d818d/photo.jpg
Server: TornadoServer/4.5.3

Access the image.

http://thumbor.example.com/image/05b2eda857314e559630c6f3334d818d/photo.jpg

So we have installed Thumbor successfully. But now if you delete the Deployment and recreate it, and you access the image path above, you will receive a 404 Not Found error.

Because we haven’t configured persistent storage for the thumbor server, we can use StatefulSet and configure PersistentVolume. But when we scale up, our image will be saved in different PVs and this will cause an error when we access the image.

We will use AWS S3 for the entire thumbor server as storage, to use S3 we follow these steps.

First, you need to create an IAM user that has permission to access the S3 bucket that you will use to store images. You follow this link https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-walkthroughs-managing-access-example1.html to create an IAM user. Then you create an access key and secret key of this IAM user and use those two values ​​to create a Kubernetes Secret, for example.

kubectl create secret generic s3-credentials --from-literal=AWS_ACCESS_KEY_ID=AKIAR4725YR2POTRRS5G --from-literal=AWS_SECRET_ACCESS_KEY=06wrpzQ1X3umftakgfsr1Tw7UqKd3yMYIrrk765d

Update a file thumbor.cm.yaml to add config so thumbor can use S3.

apiVersion: v1
kind: ConfigMap
metadata:
name: thumbor
data:
thumbor.conf: |
UPLOAD_ENABLED = True
TC_AWS_REGION = 'us-west-2'
TC_AWS_STORAGE_BUCKET = 'thumbor-test'
TC_AWS_STORAGE_ROOT_PATH = 'storage'
TC_AWS_RESULT_STORAGE_BUCKET = 'thumbor-test'
TC_AWS_RESULT_STORAGE_ROOT_PATH = 'result_storage'
STORAGE = 'tc_aws.storages.s3_storage'
UPLOAD_PHOTO_STORAGE = 'tc_aws.storages.s3_storage'
RESULT_STORAGE = 'tc_aws.result_storages.s3_storage'

You can see the full configuration at this link https://github.com/thumbor-community/aws. Above we use the configuration STORAGE = 'tc_aws.storages.s3_storage'and UPLOAD_PHOTO_STORAGE = 'tc_aws.storages.s3_storage' to configure for thumbor to use S3 as image storage.

Update a file thumbor-deployment.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
name: thumbor-server
spec:
replicas: 1
selector:
matchLabels:
app: thumbor
template:
metadata:
labels:
app: thumbor
spec:
containers:
- name: thumbor
image: minimalcompact/thumbor:6.7.5
command: ["thumbor"]
args: ["-p", "80", "-c", "/conf/thumbor.conf"]
ports:
- containerPort: 80
envFrom:
- secretRef:
name: s3-credentials
volumeMounts:
- name: thumbor-conf
mountPath: /conf
volumes:
- name: thumbor-conf
configMap:
name: thumbor

Recreate Configmap and Deployment.

kubectl apply -f thumbor.cm.yaml
kubectl apply -f thumbor-deployment.yaml

Now, we upload a file to the thumbor.

curl -i -H "Content-Type: image/jpeg" -H "Slug: photo.jpg" -XPOST http://thumbor.example.com/image --data-binary "@example.jpg"

If you go to S3 you will see your file.

We have successfully configured.

So we have learned how to configure Thumbor with S3. Thumbor is a great image processing service that we will probably use in production environments. If you have any questions or need more clarification, you can ask in the comment section below.

Leave a Comment