Python

Overview I had the opportunity to create a transparent text PDF from a PDF using Google Cloud Vision API, so this is a personal note for future reference. Below is an example of searching for simple. Background This time, we target PDFs consisting of a single page. Procedure Creating the Image Create an image to be used as the OCR target. With the default settings, the resulting image was blurry, so I set the resolution to 2x and performed position alignment considering the resolution in the process described below. ...

Overview I created a Python library for visualizing the contents of Archivematica METS files. For example, it visualizes aggregated results of processes (premis:event) performed during AIP creation, as shown below. Background In the following article, I introduced METSFlask, a web application for exploring Archivematica METS files in a human-friendly way. What I created this time is a library version of the functionality provided by METSFlask, making it easier to use outside of Flask. ...

Overview I had the opportunity to create an IIIF v3 manifest for video using iiif-prezi3, so this is a note for reference. https://github.com/iiif-prezi/iiif-prezi3 References Examples of IIIF manifest files and implementation examples using iiif-prezi3 are published in the IIIF Cookbook. Below is an example of creating an IIIF v3 manifest for video. https://iiif.io/api/cookbook/recipe/0003-mvm-video/ An implementation example using iiif-prezi3 is published at the following. https://iiif-prezi.github.io/iiif-prezi3/recipes/0003-mvm-video/ f c m c a a a h a h c a c p r o a a n n n w n w a n a r o n n n n n n d n d n n n i m f i v o o o o [ v o v n i f a _ _ = _ " a _ a t i g e s b p = b w s p s ( i . s o a { o i . a . m i c t = d g A " d d s g a a f o y e n h y t e e d n _ n = m n e . h t . d i p f a = = o i s " _ a _ f r i M n t g e ] h d i e e g a i R A a h t w d t s z s n f e n t t _ = d _ e t i [ i e s n i " h ( i m . 3 ' f s o o o : w 6 * t ( j h e t u t n d 4 * e a s i e s . r a ( 3 ( 0 h m n o m l t m c t i m b t 6 * w ( n n p p ( a e i d o o a 0 * d a o ( o e i k I o = t d r , h ) n _ i r r d e t n " i y g w n p n t s = _ e P h v = e " d o a d . " c m a t a a t w ) ) g e M a h a ( g t t n = i e n a u t n i t f e p i n c d ) t n t t v d y o ( s o o a t = i o p a = p r i : n _ n h 2 f _ s s " e m d / = b v " ) e f : ( h = a = / " o a : ) s i / i t " t " i p d s t e / d t V = h i a y . 4 , l i = p i " t i i , i 8 d i " s d v t f n d 0 A s i h : e i p . t ) , n . f t / d s i i n A . t " e : n " o u i p f , o g d t t o s i a " u a o / : x m i p , r t L a / t p i i a i a p / u 4 i / t o n i i r " f c i n g / i e ) . o o P ' c i s i o n a ] o f . o k " g . o . i / b : e a k i i a o , u b o i p o 5 t o f i k 7 A o a . / / 2 n _ k p i c r . n l / i o e 0 o a r o c 3 t n e c v k i 4 a g c o i b p } t i o d o e i = p k e o / o e b o k 0 n " / o / 0 , e 0 i r 0 n 0 k n e 3 R " 0 / d c - e 3 r i i m s - e a p v o m c n e m u v i a / - r m p / 0 v c - e l 0 i e v / u 0 d I i 0 n 3 e t d 0 c - o e e 0 h m / m o 3 r c , / - o m a m m o - n c a v m v v o n m _ i a n i - m d s f f v a e i e i n p g s d n / a t e e c g . o r a e j / s n / s c / a o a h a n n n i s n " v g / o , a h p t s / a a l " l g t a ) u e i b n " o e c ) n l h " = r , " o V o i m d _ e m o a n E n x e a r m s p _ l 1 e 0 2 3 4 " k ) b . m p 4 " , Summary Many other samples and implementation examples are also published. I hope this is helpful. ...

Overview I had the opportunity to manipulate CVAT data using Python, so this is a personal note for future reference. Setup This time, we will start it using Docker. g c d i d o t c c k c v e l a r o t n c e o m h p t o t s p e s : u / p / g - i d t h u b . c o m / c v a t - a i / c v a t - d e p t h 1 Creating an Account Access http://localhost:8080 and create an account. ...

Overview I created a sample program for performing similar image search using GUIE (Google Universal Image Embedding) pre-trained models. You can access the notebook from the following link. https://colab.research.google.com/github/nakamura196/000_tools/blob/main/guie_sample.ipynb References It uses the model output from the following notebook. https://www.kaggle.com/code/francischen1991/tf-baseline-v2-submission Usage Notes Kaggle Account A Kaggle account is required to run the notebook. Obtain a Kaggle API Key and register it in your secrets. If the following is displayed, please click “Allow access.” ...

Overview I created a library that applies Google Cloud Vision to image files and generates IIIF manifest and TEI/XML files. https://github.com/nakamura196/iiif_tei_py This article explains how to use the library. Usage You can check the usage and more at the following page. https://nakamura196.github.io/iiif_tei_py/ Installing the Library Install the library from the GitHub repository. p i p i n s t a l l h t t p s : / / g i t h u b . c o m / n a k a m u r a 1 9 6 / i i i f _ t e i _ p y Creating a GC Service Account Download a GC (Google Cloud) service account key (JSON file) by referring to articles such as the following. ...

Overview I created a library for registering RDF data to Dydra using Python. https://github.com/nakamura196/dydra-py It includes some incomplete implementations, but we hope it proves useful in some situations. Implementation Details The import is performed in the following file. https://github.com/nakamura196/dydra-py/blob/main/dydra_py/api.py#L55 It uses the SPARQL INSERT DATA operation as follows. d e f i " I A " h } f p f m " m r " e i r o p " p g " a l i r o o s d e n r r : f f g e " " s t f # g g n i e i r i e t t i o r r A C ( i r r t f l f e f l _ s l r a s u o = " l R a a _ s s s b e m p t n N e D p p d g e v p r e y R _ a h = h t s u F h h a r q I { } " : q I G } } " e p o e p : p _ D p t _ o e e m i フ . t a u N n } " u N R } } " r r n s r r f F a u { r n l b n ァ = p a p e S t " e S A " b i s p i i i t ( r i t f e イ a h r E _ r E P o n e o n n l d h s i z - . r t ル r r = _ y R d y R H { s t n t t e a t a T _ q の d s u T a T n e ( = s ( ( ( t ( r ( t y c o d 読 f e g r = t = < t : q e " f s a s ) s i p h f m み l ( r i D a D { _ u r . D " e t : t o e u ( 込 i f a f A } f A g d e e s a E l f r r n " n c f み b i p i " T " T r a r q t t r f r ) T , " : k h i . l h s " A " A a t y u a a r , o : h : _ u l G e . " " p a ) e t o m e o " r n e r , s N { { h } s u s r f T p f a d k s a e o { { _ t s u : i a h f t " p f s ) p f r n u s _ c l e o i B p _ : : h o i e r . c c { e f r o e l f ( r a : i p o e r _ i p m n a i i " ) m l } o d s e p l a a a r c l , a i > s e s s a e t t l e a e t z t f p t h ) r t ( l = e { ( = u o h i o : i f e f ( { s = l n , n t f { o i n o f e l s t o U s n l ( r o l 2 y e f o t R e / e f m r f 0 . o t h I l s _ i a m . 0 i s r t h e f p p l t a e : n t m h e o . a a e ) t n s a a e R f a r t s = d e t t R D p q h ) ' p r u , D D F t i l , ) # n o t s y F h _ - t i e _ g d f e k u f フ ' n d c r r f i e p o ォ ) t . o a a i l g y d r ー , " d p l e r } a m マ ) e h s e a " t a ッ d } _ t ( p , e t ト a u o t e h " = は t { r r o . f フ a r i e g w o ァ = e = . i . h r イ q s N m , e m ル u p o p r a に e o n o ' e t 応 r n e r x ) じ y s , t m d て , e . l a 変 . v ' t 更 h t e , a e e r a x b ' w d t o n i e } s t l r " e ' l s ) = ) = F . b h a e e l a s i d e n e ) s r : e s r ) t e d . D e f a u l t s t o N o n e . Key Design Decision One notable design decision was handling large RDF files. When uploading large RDF files all at once, there were cases where the process would stop midway. ...

Overview This is a memo about a library I created for generating RDF files from VSDX files. https://github.com/nakamura196/vsdx-rdf Background I have been exploring methods for creating RDF data using Microsoft Visio in articles like the following. This article corresponds to the note in the above article that said “This library will be introduced in a separate article.” Usage Please refer to the following. https://nakamura196.github.io/vsdx-rdf/ Google Colab A notebook is available for trying out this library. ...

Here is a script for fetching all records from an OAI-PMH repository using Python. I hope it serves as a useful reference. i f i # b # p } r # r p p r d # w p p m r m a a e e r r o a h r r p o p D s I r s P q e i o t F i i i o m o e e n a p r p n t a e l n n r r f _ i m ' ' o e = a t t e # f # t i p } r r t t t r t i u t s v m s n p r ( = = c o o f a e o ( ( e n r i e e e s a R e " h T P r G k r s o " l r q x e l a = r t t e r e d S E [ r r e e t a p t A e e u m l b a ' e q _ e T ] a u o r i p # d t n o b m ' o l n q e l t = { ' d : = u r n . l e c e d r a _ k r s v r n = l ( u s . h O : a t e e d f l : e c e i O t r e e e e e s d e t e e ' A t ' r h s q i r s o n n t a e l n a = r s e E r a s s t h I ' a 9 e e t n o d s r t t h . s e _ k b u T e t t r e t - L P 7 q ( = g m a d i ( e a u m e { ' m = . c a s i e n t P i r 5 u i ' s t r f f r p m e l : p f o ) m e d p M s e 9 e n G r r t a e i i ' p p n e # t r r r ) p . p s H t f 0 s i E e e r c n e R d e t t m ' i e o d o E o : R i ' t t T q q i o r e a n i e E L o q m s r l i / r e x s i ' . u n r r c t d o = n n i n u s t e n / e c ' . a , p e g d o = o a ( n t d s T e t h m t c q o : g l r s ( s o r r T r t o s r a R e u u r e b e t r t r d c e o o i l R k t i v e n r e d ' t r a p e . e a c k o s o e e s n e q t a s s c ( e s a t s f c I n o e t o c n . g u T t t ' u b q e r o p i o D r n . N p o ' g ( b e r i , r a u _ e : o n r : b d f o r : e r e s e o a s e u ( " n d d e ) a i n i d t e e t e n t e s r ) , s a . { n n e f s t ( s n . i _ t l e l f i p d d ' o b p a l o u , p . l i d r ( o n , k a o f s i n r p r c ( n e o e ' r o e s n e b ' l a e o ' d n c x . n e s t E r , , r p n . ( t e e / n t _ _ e c T a a a t / ' i s c / o o e u . h r p m r e . f s u { t k l r c e y a s e n { / i e t h e e l o d . r = d t h / e d e t t n m , n . t a p _ ) t { r t o e t " . m a r t h } h n p k n p e ) u s r e p t ' e e : e t a n - = a q : t ) r x / n . r t t p m . / p e t / _ t a ) o a s u / : w e e m k r ) r w / a r w l x s y a l w / s e w e t = o m ) w w q . m p . s . w w u o e a a ) o w e e p n r c # p . l s e t a . e o l t n . m j O n p a t s p u a e r e ) / t r n c x o p c a h t a u h r i : i t i c v ' v h e t e i s h s v . e . e o o s r U r . g R g / L r O O g A A / I I O / / A 2 2 I . . / 0 0 2 / / . } } 0 r r / e e } s c i u o d m r e p d n t ' t i ) i o : f n i T e o r k ' e ) n . ' t ) e x t

Overview I had the opportunity to bulk delete multiple content items using the Drupal REST API, so this is a memo of the process. References For a method to bulk delete content without using the REST API, please refer to the following. Preparation First, enable the HTTP Basic Authentication module and the JSON:API module. Additionally, enable DELETE in REST resources. /admin/config/services/rest Execution Example The following custom library is used. ...

Overview I prototyped an app using Zotero’s API and Streamlit. https://nakamura196-zotero.streamlit.app/ This article is a memo on developing this app. Streamlit The following article was very helpful. https://qiita.com/sypn/items/80962d84126be4092d3c Zotero’s API Zotero’s API is described at the following page. https://www.zotero.org/support/dev/web_api/v3/start This time, I used the following library introduced on the above page. https://github.com/urschrei/pyzotero To use the API, you need to obtain a personal library ID and an API key, which could be obtained by following the Quickstart steps in the README. ...

Overview I had an opportunity to retrieve RDF data from Wikidata entity URIs, so here are my notes. Without Content Negotiation First, make a request with empty headers as follows. i # u h } # r # i e m r e e f l p U l a S s C s o R d e p h r e r L = e n o e e t p : p t r d n c s e r r f " s i s k p x i i r o h n e i o t n n e r t = g n n t t q t = g s = ( ( u t p { a e t " e h : r i . r e F s e / G e f s e x a t / E q t s t i s W w T u t a p [ l i w e h t o : e k w r s e u n 5 d i . e t s s 0 d w q s r _ e 0 t a i u . e c . 0 o t k e g q o t ] a i s e u d e ) r d t t e e x e e a ( s t t n t t u t = r t a o r = i i . l w e t o t , a 2 v y r h s 0 e g e h 0 i / e s : R n e U a u D n R d c F R t L e c D i r e d F t s s a y = s t f h f a Q e u . r 1 a l m 2 d S a 4 e t t 1 r a 8 s t " ) u s c o d e : " , r e s p o n s e . s t a t u s _ c o d e ) In this case, you can retrieve text data in JSON format as follows. ...

Overview As IIIF Presentation API 3 becomes more widespread, I found it increasingly difficult to understand the specification and create JSON files directly. So I tried using the following Python library, and this is a note for reference. https://github.com/iiif-prezi/iiif-prezi3 I used this library for converting the data published on the Toji Hyakugo Monjo WEB to IIIF, as introduced in the following article. The source code may be hard to read, but it is also published in the following repository, and I hope it is helpful. ...

Overview I had the opportunity to use “ARC2 RDF Graph Visualization” published by Masahide Kanzaki from Python, so here are my notes. The public page for “ARC2 RDF Graph Visualization” is below. https://www.kanzaki.com/works/2009/pub/graph-draw By providing RDF described in Turtle, RDF/XML, JSON-LD, TriG, or Microdata as input, you can obtain visualization results as png or svg files. Usage Example in Python i t o # u d } # r # i e m e u r a e f l p x t D l t S s C s o t p a a e p h r e r u t = " " " " " n o e e p # : o # w t = t a = R r g r q d n c s r s i _ " D t t a n s k p i p . S t r " p n h { F y y n a P e o n r m a h e @ a e t " p p k m O i n t i a v f q p t e t : e e d e S = f s ( n k e o . u r h d p " " i " T e " t e p w e e e s t : : r : r r . R ( d r e r s f = d : e " r e e h e r i e n i t i / x " " : " e q s e s e r s ( t s x " f t t p o q u p a p s s p o e . w , u n " n u e o d o p ( o u ( n / r w r g l " e s n e n o o n t r s g w t " r , s t s r s n s s p e 1 r P . l , " t s e s e s . e u s : a O k e , . [ e p t p p S a " p i ' i . a a _ o < h T n , o s C s t t s p n h . z s o e h a s t p r a t n n n x . P t e t n e k ( o t o t d N h . p g q i u t e t [ i G , c : " u . r n : r o / e c l a t a 5 n f ' n / s o , - 0 a i w t e t m P T P 0 m l b e x / d N y N ] e e ' n a w a G p G ) ( ) t m o t e o ) p r a i ' i u a l k = m ] m # t s e s d a a p . / a g ! g D u f o 2 t e = e i t : r 0 a . s _ g 0 ) ' p p / 9 i D l a p / m i a t r p a s y h o u g p ) p b e l f , e / / a i r g p y r e y r n i s x / a g n t i > p ' g s h : 5 t - c 0 _ \ d o 0 o n r n k \ a t c = n w e h T < " n a r h t r u t : a e t " c ) p ) t : e / r / s e x # a m [ p : l 5 e 0 . 0 o ] r g / b b b > n s 1 : a a a \ " c c c \ " . " Summary I hope this is helpful for visualizing RDF data. ...

Overview In the following article, I introduced an inference app using Hugging Face Spaces and a YOLOv5 model trained on the NDL-DocL dataset. This app had stopped working, so I fixed it to make it operational again. https://huggingface.co/spaces/nakamura196/yolov5-ndl-layout Here are my notes on the changes made during this fix. Changes The modified app.py is shown below. i f i i m d i o ] t d a e ] d d m r m m o e n u i e r x e e p o p p d f p t t s t a m m o m o o e u p l c i m o o r r r l y r d r i # o r ] t u g g e r c p [ [ [ . t P t t o e f e m u e s t r r i l l ' ' ' = l I = l s s _ C t t s . . = p e e 『 『 『 a g L y j o u = w o p u = I J t s 源 源 平 g u r o s y ( l = i n u r o r = m S " i = 氏 氏 家 r n a i l o o i t r t v t n u e g a O Y o = 物 物 物 . c d m o n l m s e j h e _ t s r [ g N O n " 語 語 語 I h i p v o ) s s _ r i [ p . e ( L < [ 』 』 』 n ( o o 5 v : = u o b t m u I ( ) O = p ( ( ( t s r 5 l n o a t m t v 東 京 国 e h a t . m t . x t g _ a y 5 " s 京 都 文 r a s l o s l e h e i g p Y t 大 大 学 f r I o d . o s e m e e N O y 学 学 研 a e g m a e p a = a ( = D L l 総 所 究 c = r a d l a d = n g t " L O e 合 蔵 資 e F g ( ( n s u I e y p - v = 図 ) 料 ( a e " i d ( r m m , p i D 5 ' 書 . 館 y l n m a d e p a e l o t 館 j 提 o s a ) s f s y g = " c N e 所 p 供 l e k ( ) u e ' , L D x 蔵 g ) o ) a ) l a . p L t ) ' . , m # . t r f i l D - - . ] j u x s r r l a a D a j , p i r i y . a o ' b t o l p g n a n x r y m , e a c i g ' p 1 f y e a l s L g ' ] u 9 e [ n b r l = e n ] t 6 r 0 d a r a " t D : , s / e ] e c a b O s a , y n . r k y e u " t c o c t ( ( l t a e o l e o ) t i = p s n u o _ [ o m " u e t t v j 0 _ O t t e p 5 s ] a w r s r u - o n i i I ' t n n t g m G > s d ( # i h i a r Y , l o m _ n g a O - r r a b a e d L t l i e g o l " i O i a e s e x ) o v t y n u e I , 5 l o t l s m d e u = t ) a e N = t " s g m D t " r . e o L i ) e r " - t c e ) f D l o n o o e r d r c , d e L s r o d " ( b D e ) ) j a s e t c r c a r e t s i t e p u d t t r e s i n t o s e i n c s = a t d i a e l o n s i n c s . o r t b i U j p o p e t f l c i o t o i a n m d d , a e g a t a e n e r s c t i t i m i c a o l g n e e = m a o o r r d t e i c l c l l i t e c r , k a i e a n x n e a d m e p x o l a n e m s p t = l h e e e x a i < m m a p a l g h e e r s e ) t f = u " s h e t . t " p s : / / g i t h u b . c o m / n d l - l a b / l a y o u t - d a t a s e t \ " > N D L - D o c L D a t a s e t s < / a > . < / p > " First, due to Gradio version upgrades, I changed gr.inputs.Image to gr.Image and similar updates. ...

Overview Notes on programs for formatting XML strings in Python. Program 1 I referenced the following. https://hawk-tech-blog.com/python-learn-prettyprint-xml/ I added processing to remove unnecessary blank lines. f i d r m e o p f m o r p r p p p r x t r e r r r e m e p e e e t l r t a t t t u . e t r t t t r d i s y y y n o f e m y d = = = p ( r i r = r p r e m o e r e t p u m . e . t o g i s t s y r h n u t u t _ i b y b s d ( . ( m t o r r r i r m " e " n i . [ p \ i n p l n d g a t a \ o ) r c s m : s ] e * e + ( \ S \ " n t n > " r " \ , i , n n " g " n ( " \ n r , t " o < , u r " g e , p h p r _ a " e s r > t t s \ t r e n y i d \ ) n . t g t < ) o " # p ) r R e e t # p t l y R a x e c m m e l o ( v c i e o n n d u s e n e n n c t e u = c t " e i \ s v t s e " a ) r l ) y i n b e # l a b R n r e k e m a o l k v i s e n e ( u s i n n n c e l c u e d s i s n a g r y b l l a i n n k e l b i r n e e a s k ) s w a i f t t h e r a i s n i d n e g n l t e a t l i i o n n e b r e a k Program 2 I referenced the following. ...

Overview “SAT Daizokyo Text Database 2018” is described as follows. https://21dzk.l.u-tokyo.ac.jp/SAT2018/master30.php This site is the 2018 version of the digital research environment provided by the SAT Daizokyo Text Database Research Society. Since April 2008, the SAT Daizokyo Text Database Research Society has provided a full-text search service for all 85 volumes of the text portion of the Taisho Shinshu Daizokyo, while enhancing usability through collaboration with various web services and exploring the possibilities of web-based humanities research environments. In SAT2018, we have incorporated new services including collaboration with high-resolution images via IIIF using recently spreading machine learning technology, publication of modern Japanese translations understandable by high school students with linkage to the original text. We have also updated the Chinese characters in the main text to Unicode 10.0 and integrated most functions of the previously published SAT Taisho Image Database. However, this release also provides a framework for collaboration, and going forward, data will be expanded along these lines to further enhance usability. The web services provided by our research society rely on services and support from various stakeholders. For the new services in SAT2018, we received support from the Institute for Research in Humanities regarding machine learning and IIIF integration, and from the Japan Buddhist Federation and Buddhist researchers nationwide for creating modern Japanese translations. We hope that SAT2018 will be useful not only for Buddhist researchers but also for various people interested in Buddhist texts. Furthermore, we would be delighted if the approach to applying technology to cultural materials presented here serves as a model for humanities research. ...

Overview I had the opportunity to create a publication list using the researchmap API, so here are my notes. Query Examples for the researchmap API Here are some query examples for the researchmap API. Retrieve a list of papers https://api.researchmap.jp/nakamura.satoru/published_papers Specify a limit (limit usage) https://api.researchmap.jp/nakamura.satoru/published_papers?limit=5 Retrieve results from a specific offset (start usage) https://api.researchmap.jp/nakamura.satoru/published_papers?limit=5&start=6 Specify publication dates (from_date and to_date) https://api.researchmap.jp/nakamura.satoru/published_papers?from_date=2023-04-01&to_date=2024-03-31 Python Usage Example Based on the specified user and publication dates, export published_papers and presentations to Excel. ...

Overview In this article, I try out the following tool. https://github.com/artefactual-labs/AIPscan This tool is described as follows: AIPscan was developed to provide a more in-depth reporting solution for Archivematica users. It crawls METS files from AIPs in the Archivematica Storage Service to generate tabular and visual reports about repository holdings. It is designed to run as a stand-alone add-on to Archivematica. It only needs a valid Storage Service API key to fetch source data. ...

Overview “Digital Genji Monogatari” is a site that aims to propose an environment to support research on The Tale of Genji as well as education and research activities using classical texts, by collecting and creating various related data about The Tale of Genji and linking them together. https://genji.dl.itc.u-tokyo.ac.jp/ One of the features provided by this site is the “alignment of the Collated Tale of Genji with modern Japanese translations.” As shown below, the corresponding sections between the “Collated Tale of Genji” and Yosano Akiko’s translation published on Aozora Bunko are highlighted. ...