tubesync/docs/other-database-backends.md

TubeSync

Advanced usage guide - using other database backends

This is a new feature in v1.0 of TubeSync and later. It allows you to use a custom existing external database server instead of the default SQLite database. You may want to use this if you encounter performance issues with adding very large or a large number of channels and database write contention (as shown by errors in the log) become an issue.

Requirements

TubeSync supports SQLite (the automatic default) as well as PostgreSQL, MySQL and MariaDB. For MariaDB just follow the MySQL instructions as the driver is the same.

You should start with a blank install of TubeSync. Migrating to a new database will reset your database. If you are comfortable with Django you can export and re-import existing database data with:

# Stop services
$ docker exec -t tubesync \
    bash -c 'for svc in \
    /run/service/{gunicorn,tubesync*-worker} ; \
do \
    /command/s6-svc -wd -D "${svc}" ; \
done'
# Backup the database into a compressed file
$ docker exec -t tubesync \
    python3 /app/manage.py \
    dumpdata --format jsonl \
    --exclude background_task \
    --output /downloads/tubesync-database-backup.jsonl.xz

Writing the compressed backup file to your /downloads/ makes sense, as long as that directory is still available after destroying the current container. If you have a configuration where that file will be deleted, choose a different place to store the output (perhaps /config/, if it has sufficient storage available) and place the file there instead.

You can also copy the file from the container to the local filesystem (/tmp/ in this example) with:

$ docker cp \
    tubesync:/downloads/tubesync-database-backup.jsonl.xz \
    /tmp/

If you use - as the destination, then docker cp provides a tar archive.

After you have changed your database backend over, then use:

# Stop services
$ docker exec -t tubesync \
    bash -c 'for svc in \
    /run/service/{gunicorn,tubesync*-worker} ; \
do \
    /command/s6-svc -wd -D "${svc}" ; \
done'
# Load fixture file into the database
$ docker exec -t tubesync \
    python3 /app/manage.py \
    loaddata /downloads/tubesync-database-backup.jsonl.xz

Or, if you only have the copy in /tmp/, then you would use:

# Stop services
$ docker exec -t tubesync \
    bash -c 'for svc in \
    /run/service/{gunicorn,tubesync*-worker} ; \
do \
    /command/s6-svc -wd -D "${svc}" ; \
done'
# Load fixture data from standard input into the database
$ xzcat /tmp/tubesync-database-backup.jsonl.xz | \
    docker exec -i tubesync \
    python3 /app/manage.py \
    loaddata --format=jsonl -

As detailed in the Django documentation:

https://docs.djangoproject.com/en/5.1/ref/django-admin/#dumpdata

and:

https://docs.djangoproject.com/en/5.1/ref/django-admin/#loaddata

Further instructions are beyond the scope of TubeSync documenation and you should refer to Django documentation for more details.

If you are not comfortable with the above, then skip the dumpdata steps, however remember you will start again with a completely new database.

Steps

1. Create a database in your external database server

You need to create a database and a user with permissions to access the database in your chosen external database server. Steps vary between PostgreSQL, MySQL and MariaDB so this is up to you to work out.

2. Set the database connection string environment variable

You need to provide the database connection details to TubeSync via an environment variable. The environment variable name is DATABASE_CONNECTION and the format is the standard URL-style string. Examples are:

postgresql://tubesync:password@localhost:5432/tubesync

and

mysql://tubesync:password@localhost:3306/tubesync

Important note: For MySQL databases make SURE you create the tubesync database with utf8mb4 encoding, like:

CREATE DATABASE tubesync CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;

Without utf8mb4 encoding things like emojis in video titles (or any extended UTF8 characters) can cause issues.

3. Start TubeSync and check the logs

Once you start TubeSync with the new database connection you should see the folling log entry in the container or stdout logs:

2021-04-04 22:42:17,912 [tubesync/INFO] Using database connection: django.db.backends.postgresql://tubesync:[hidden]@localhost:5432/tubesync

If you see a line similar to the above and the web interface loads, congratulations, you are now using an external database server for your TubeSync data!

Database Compression (For MariaDB)

With a lot of media files the sync_media table grows in size quickly. You can save space using column compression using the following steps while using MariaDB:

1. Stop tubesync
2. Execute ALTER TABLE sync_media MODIFY metadata LONGTEXT COMPRESSED; on database tubesync
3. Start tunesync and confirm the connection still works.

Docker Compose

If you're using Docker Compose and simply want to connect to another container with the DB for the performance benefits, a configuration like this would be enough:

 tubesync-db:
  image: postgres:17
  container_name: tubesync-db
  restart: unless-stopped
  volumes:
   - /<path/to>/tubesync-db:/var/lib/postgresql/data
  environment:
   - POSTGRES_DB=tubesync
   - POSTGRES_USER=postgres
   - POSTGRES_PASSWORD=testpassword

 tubesync:
  image: ghcr.io/meeb/tubesync:latest
  container_name: tubesync
  restart: unless-stopped
  ports:
   - 4848:4848
  volumes:
   - /<path/to>/tubesync/config:/config
   - /<path/to>/YouTube:/downloads
  environment:
   - DATABASE_CONNECTION=postgresql://postgres:testpassword@tubesync-db:5432/tubesync
  depends_on:
   - tubesync-db