Overview
This is a memo about hosting TEI/XML files on S3-compatible object storage. Specifically, we target the mdx I object storage.
https://mdx.jp/mdx1/p/about/system
Background
We are building a web application (Next.js) that loads TEI/XML files and visualizes their content. When the number and size of files were small, they were stored in the public folder, but as these grew larger, we considered hosting them elsewhere.
There are many options for storage locations, but this time we target mdx I’s S3-compatible object storage.
Uploading Files to Object Storage via GUI
There are many ways to upload TEI/XML files to object storage via GUI. Among them, I have previously introduced methods using Cyberduck and GakuNin RDM.
In this case, however, content other than TEI/XML was managed in Drupal. Therefore, we connected Drupal to the object storage so that users could complete everything through Drupal operations.
Connecting Drupal to Object Storage
The following module is used.
https://www.drupal.org/project/s3fs
After installation, select S3 File System from the configuration page /admin/config.
Then register the access key and secret key, along with the S3 bucket name.
Also, in Advanced Configuration Options under Custom Host Settings, enter https://s3ds.mdx.jp.
This completes the connection settings to the object storage.
After that, in the feed settings for each content type, select “S3 File System” as the upload destination.
Also, since TEI/XML files are the upload target in this case, enter xml as the “Allowed extensions.”
As a result, TEI/XML files uploaded through Drupal’s GUI are now stored in mdx I’s object storage.
(Reference) Bulk File Upload Using Drupal’s JSON:API
For the initial registration of TEI/XML files, bulk registration was performed using Python. The following article was helpful for bulk file upload methods using JSON:API.
https://www.drupal.org/node/3024331
As an example, it was achieved with a script like the following.
This can be used to upload files to a field such as field_file on content that has already been created.
There may be more appropriate methods, but it can be used as follows.
The following environment variables are required.
Using from Next.js
TEI/XML files uploaded to the object storage are loaded from applications such as Next.js.
The following library was used successfully.
https://www.npmjs.com/package/@aws-sdk/client-s3
Specifically, it was used as follows.
It is used with the following environment variables.
As a result, the following architecture can be achieved.
Future Outlook: Connecting with LEAF-Writer
While we have not prepared an editing environment for TEI/XML files uploaded to Drupal (or rather, mdx I’s object storage) in this case, using the following LEAF-Writer Drupal module could potentially allow TEI/XML file editing and management to be completed within the CMS.
https://gitlab.com/calincs/cwrc/leaf-writer/leaf_writer
Also, the following prototype connecting GakuNin RDM and LEAF-Writer may be a useful reference.
Summary
This article introduced an example of hosting TEI/XML files on S3-compatible object storage. Since there are both advantages and disadvantages to this approach, I hope this article serves as a useful reference when considering architectures suited to your use case.