python – Configure Django and Google Cloud Storage?

May, 2022 Update:

With this instruction, you can connect your Django app to your bucket on GCS(Google Cloud Storage) and you can serve your static files and serve, upload and delete your media files.

For example, you have the bucket “my-django-bucket” on GCS:

And you have the service account “my-django-bucket-sa” then you need to copy(Ctrl+C) the email “my-django-bucket-sa@myproject-347313.iam.gserviceaccount.com”:

enter image description here

Next, in Bucket details of the bucket “my-django-bucket”click on “PERMISSIONS” then “ADD”:

enter image description here

Then, to enable the service account “my-django-bucket-sa” to have the full control of GCS resourcespaste(Ctrl+V) the email “my-django-bucket-sa@myproject-347313.iam.gserviceaccount.com” to “New principals” then choose the role “Storage Admin” then click on “SAVE”. *Choose other role by checking IAM roles for Cloud Storage if you don’t want the role “Storage Admin” which can have the full control of GCS resources:

enter image description here

Next, to enable all users to view(read) files, type “allUsers” to “New principals” then choose the role “Storage Legacy Object Reader” then click on “SAVE”:

enter image description here

Then, you should be asked as shown below so click on “ALLOW PUBLIC ACCESS”:

enter image description here

Finally, you could add the role “Storage Admin” to the service account “my-django-bucket-sa” and the role “Storage Legacy Object Reader” to “allUsers”:

enter image description here

Next, you need to download the private key of the service account “my-django-bucket-sa” in JSON so click on “Manage keys” from the 3 dots “⋮”:

enter image description here

Then, click on “Create new key” from “ADD KEY”:

enter image description here

Then, choose “JSON” then click on “CREATE”

enter image description here

Finally, you could download the private key of the service account “my-django-bucket-sa” in JSON “myproject-347313-020754294843.json”:

enter image description here

Now, you have a django project and there are one settings folder “core” which has “static/core/core.js” and “settings.py” and one application folder “myapp” which has “static/myapp/myapp.css” as shown below:

enter image description here

Then next, you need to put “myproject-347313-020754294843.json” to the root django project directory where “db.sqlite3” and “manage.py” are:

enter image description here

Then, you better rename “myproject-347313-020754294843.json” to a short and reasonable name such as “gcpCredentials.json”:

enter image description here

Next, you need to install django-stores[google]” to connect to and communicate with “my-django-bucket” on GCS(Google Cloud Storage):

pip install django-storages[google]

By installing django-stores[google]”you can get “django-stores” and other necessary packages as shown below:

"requirements.txt"

django-storages==1.12.3
cachetools==4.2.4
google-api-core==2.7.2
google-auth==2.6.5
google-cloud-core==2.3.0
google-cloud-storage==2.0.0
google-crc32c==1.3.0
google-resumable-media==2.3.2
googleapis-common-protos==1.56.0
protobuf==3.19.4
pyasn1==0.4.8
pyasn1-modules==0.2.8

Be careful, if you install “django-stores” without “[google]” as shown below:

pip install django-storages

You can only get “django-stores” as shown below.

"requirements.txt"

django-storages==1.12.3

Next, create “gcsUtils.py” in “core” folder where “settings.py” is:

enter image description here

Then, put this code below to “gcsUtils.py” to define “Static” and “Media” variables which each have a “GoogleCloudStorage” class object:

# "core/gcsUtils.py"

from storages.backends.gcloud import GoogleCloudStorage

Static = lambda: GoogleCloudStorage(location='static')
Media = lambda: GoogleCloudStorage(location='media')

Next, add this code below to “settings.py”. *“STATICFILES_STORAGE” is like the conbination of “STATIC_ROOT” and “STATIC_URL” and “DEFAULT_FILE_STORAGE” is like the conbination of “MEDIA_ROOT” and “MEDIA_URL”:

# "core/settings.py"

from google.oauth2 import service_account

# Set "static" folder
STATICFILES_STORAGE = 'core.gcsUtils.Static'

# Set "media" folder
DEFAULT_FILE_STORAGE = 'core.gcsUtils.Media'

GS_BUCKET_NAME = 'my-django-bucket'

# Add an unique ID to a file name if same file name exists
GS_FILE_OVERWRITE = False

GS_CREDENTIALS = service_account.Credentials.from_service_account_file(
    os.path.join(BASE_DIR, 'gcpCredentials.json'),
)

Then, run this command below:

python manage.py collectstatic

Now, “static” folder is created in “my-django-bucket”:

enter image description here

And static files are collected from “admin” and “application” folders to “static” folder in “my-django-bucket”:

enter image description here

And this is “myapp.css” in “myapp” folder:

enter image description here

But as you can see, static files are not collected from the settings folder “core” to “static” folder in “my-django-bucket”:

enter image description here

Because “STATICFILES_STORAGE” can only collect static files from “admin” and “application” folders but not from other folders like the settings folder “core”:

# "core/settings.py"

STATICFILES_STORAGE = 'core.gcsUtils.Static'

So, to collect static files from the settings folder “core” to “static” folder in “my-django-bucket”:

enter image description here

You need to add “STATICFILES_DIRS” to “settings.py” as shown below:

# "core/settings.py"

# Collect static files from the settings folder
# "core" which is not "admin" and "application" folder
STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'core/static'),
]

Then, this is the full code of “settings.py”:

# "core/settings.py"

from google.oauth2 import service_account

# Collect static files from the settings folder
# "core" which is not "admin" and "application" folder
STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'core/static'),
]

# Set "static" folder
STATICFILES_STORAGE = 'core.gcsUtils.Static'

# Set "media" folder
DEFAULT_FILE_STORAGE = 'core.gcsUtils.Media'

GS_BUCKET_NAME = 'my-django-bucket'

# Add an unique ID to a file name if same file name exists
GS_FILE_OVERWRITE = False

GS_CREDENTIALS = service_account.Credentials.from_service_account_file(
    os.path.join(BASE_DIR, 'gcpCredentials.json'),
)

Then again, run this command below:

python manage.py collectstatic

Then, static files are collected from the settings folder “core” to “static” folder in “my-django-bucket”:

enter image description here

And this is “core.js” in “core” folder:

enter image description here

Next, this is the code for “myapp/models.py”:

# "myapp/models.py"

from django.db import models

class Image(models.Model):
    image = models.ImageField(upload_to='images/fruits')
    
    def  __str__(self):
        return  str(self.image)

And this is the code for “myapp/admin.py”:

# "myapp/admin.py"

from django.contrib import admin
from .models import Image

admin.site.register(Image)

Then, upload “orange.jpg”:

enter image description here

Now, “media” folder is created in “my-django-bucket”:

enter image description here

And “orange.jpg” is uploaded in “media/images/fruits”:

enter image description here

And because “GS_FILE_OVERWRITE = False” is set in “settings.py”:

# "core/settings.py"

# Add an unique ID to a file name if same file name exists
GS_FILE_OVERWRITE = False

If uploading the same name file “orange.jpg” again:

enter image description here

Then, the unique ID “_VPJxGBW” is added to “orange.jpg” to prevent file overwrite then, “orange_VPJxGBW.jpg” is uploaded as shown below:

enter image description here

Next, if there is “orange.jpg” in “media/images/fruits”:

enter image description here

Then, update(change) “orange.jpg” to “apple.jpg” being uploaded:

enter image description here

Then, “apple.jpg” is uploaded in “media/images/fruits” but “orange.jpg” is still in “media/images/fruits” without deleted:

enter image description here

And if there is “orange.jpg” in “media/images/fruits”:

enter image description here

Then, delete “orange.jpg”:

enter image description here

But “orange.jpg” is still in “media/images/fruits” without deleted:

enter image description here

So, to delete uploaded files when they are updated(changed) and deleted, you need to install “django-cleanup”:

pip install django-cleanup

Then, add it to the bottom of “INSTALLED_APPS” in “settings.py”:

# "core/settings.py"

INSTALLED_APPS = (
    ...,
    'django_cleanup.apps.CleanupConfig', # Here
)

Then, if there is “orange.jpg” in “media/images/fruits”:

enter image description here

Then, update(change) “orange.jpg” to “apple.jpg” being uploaded:

enter image description here

Then, “apple.jpg” is uploaded in “media/images/fruits” and “orange.jpg” is deleted from “media/images/fruits”:

enter image description here

And if there is “orange.jpg” in “media/images/fruits”:

enter image description here

Then, delete “orange.jpg”:

enter image description here

Then, “orange.jpg” is deleted from “media/images/fruits”:

enter image description here

lastly, the GCS Bucket Settings which you have just set in “settings.py” as shown below work in both “DEBUG = True” and “DEBUG = False”:

# "core/settings.py"

from google.oauth2 import service_account

# Collect static files from the settings folder
# "core" which is not "admin" and "application" folder
STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'core/static'),
]

# Set "static" folder
STATICFILES_STORAGE = 'core.gcsUtils.Static'

# Set "media" folder
DEFAULT_FILE_STORAGE = 'core.gcsUtils.Media'

GS_BUCKET_NAME = 'my-django-bucket'

# Add an unique ID to a file name if same file name exists
GS_FILE_OVERWRITE = False

GS_CREDENTIALS = service_account.Credentials.from_service_account_file(
    os.path.join(BASE_DIR, 'gcpCredentials.json'),
)

And, there will be “STATIC_ROOT”, “STATIC_URL”, “MEDIA_ROOT”, “MEDIA_URL” with the GCS Bucket Settings in “settings.py” as shown below. So, in this case, “STATIC_ROOT”, “STATIC_URL”, “MEDIA_ROOT” and “MEDIA_URL” don’t work while the GCS Bucket Settings work communicating with “my-django-bucket” on GCS:

# "core/settings.py"

from google.oauth2 import service_account

STATIC_ROOT = os.path.join(BASE_DIR, 'static')
STATIC_URL = '/static/'
  
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'

'''GCS Bucket Settings Start'''
# Collect static files from the settings folder
# "core" which is not "admin" and "application" folder
STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'core/static'),
]

# Set "static" folder
STATICFILES_STORAGE = 'core.gcsUtils.Static'

# Set "media" folder
DEFAULT_FILE_STORAGE = 'core.gcsUtils.Media'

GS_BUCKET_NAME = 'my-django-bucket'

# Add an unique ID to a file name if same file name exists
GS_FILE_OVERWRITE = False

GS_CREDENTIALS = service_account.Credentials.from_service_account_file(
    os.path.join(BASE_DIR, 'gcpCredentials.json'),
)
'''GCS Bucket Settings End'''

So, if you want “STATIC_ROOT”, “STATIC_URL”, “MEDIA_ROOT” and “MEDIA_URL” to work, just comment the GCS Bucket Settings then set “STATICFILES_DIRS” as shown below:

# "core/settings.py"

from google.oauth2 import service_account

# Collect static files from the settings folder
# "core" which is not "admin" and "application" folder
STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'core/static'),
]

STATIC_ROOT = os.path.join(BASE_DIR, 'static')
STATIC_URL = '/static/'
  
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'

"""
'''GCS Bucket Settings Start'''
# Collect static files from the settings folder
# "core" which is not "admin" and "application" folder
STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'core/static'),
]

# Set "static" folder
STATICFILES_STORAGE = 'core.gcsUtils.Static'

# Set "media" folder
DEFAULT_FILE_STORAGE = 'core.gcsUtils.Media'

GS_BUCKET_NAME = 'my-django-bucket'

# Add an unique ID to a file name if same file name exists
GS_FILE_OVERWRITE = False

GS_CREDENTIALS = service_account.Credentials.from_service_account_file(
    os.path.join(BASE_DIR, 'gcpCredentials.json'),
)
'''GCS Bucket Settings End'''
"""

Leave a Comment