Overview

I had the opportunity to run a local LLM using mdx.jp’s 1GPU pack and Ollama, so this is a memo of the process.

https://mdx.jp/mdx1/p/guide/charge

References

I referred to the following article.

https://highreso.jp/edgehub/machinelearning/ollamainference.html

Downloading the Model

Here, we target llama3.1:70b.

After the download is complete, it becomes selectable as shown below.

Usage Example

We use the following “Shibusawa Eiichi Biographical Materials.”

https://github.com/shibusawa-dlab/lab1

Using the API

Documentation was found at the following location.

https://docs.openwebui.com/api/

Issue an API key separately from the JWT token as shown below.

Here is an execution example.

iit朝Auh}td}r#ppmme来Prehaerrppx少IlartsDiiootシKdeapinnrrクE=e""s""]ostttt=風YrAC=mmnp((邪"suo=oeslrrrj気=htn{dseaeeesナt=ht3es{}yssqo'ル"toe0la=ppun六ニsp{rn0"gtooe月ヨk:it:erhnns十リ-/z-s""eesst四晏x/aT""rcqees日起xltyl:oour..、xoipllneesj日xcoea[etssts　八"an"m"etpao晴時l":a:nsotn　h:3t.nu(風洗o".""pss)ナ面sfa1u:oe_)クヲt"p:sscシナ:Bp7eftoテス8el0r"(d暑、0aib"次ue気後8rc",のr)昨、0ea,テl日六/rtキ,ニ孫aiス比王p{oトhシ宮iAnはeテノ/P/渋a少神cIj沢dク官hKs栄e加又aEo一rフハtYnのsル同/}"日=ヲ志c"記h覚社o,のeフ員m一ap部d安lでe藤eすrt。s氏iテ,等oキ来nスd訪sトaス"をt、{at=十hj時rseo大sn阪}.支字d店程u長度mにp野要s口約(しd、てaくt神だa戸さ)い)杉。田要約、文の名み古を屋改行清せ水ずに及返し西て京くだ支さ店い長。句中読川点を、適其宜他使用小し林てく、ださ片い野。\、n\前n原{t等eノx諸t氏}ヲ"伴ヒ嵐山ニ抵リ、三軒屋ニテ午飧シ、船ヲ浮ヘテ大江川《（堰）》ヲ遡ル、船中囲碁ノ興アリ、嵐山ノ緑葉少シク繁茂ニ過ルモ両岸ニハ山花咲乱レテ頗ル風致アリ、午後四時過玉川楼ニ帰宿ス、今朝、尾崎司法大臣ノ秘書黒田氏来ル、又、林和太郎氏（桂氏ノ父）来話ス、午前十時ヨリ各支店主任ト共ニ嵐山ニ抵リ三軒屋ニテ午飧ス、後、大江川《（大堰川）》ニ船ヲ浮ヘ、午後四時帰宿ス、後、玉川楼ニ於テ晩飧会ヲ開ク、種々ノ余興アリ、夜十時散会ス、中井三郎兵衛氏モ来会ス'''

As a result, the following was obtained. However, it took nearly 60 seconds for the result to be returned.

{}']i'''{},dcmc'}o'rohi''''b:edonlfmc'jaeidoieore'tlcegnsnocle'expistltld:s'rsaee'a'':ohgn':m:':b_et:al0sr'''31l[,'e::'c.7a:aah13ms{'sa0aNo六st:63on月i.7.n'十sc711e:四to01,日amb7:'はnp-3s晴tl4,7tれ'e40oでt2bp風i5''がo4,,なnfく'4て6暑-か0っfた3。f朝-は4少0しc風1邪-気9が3あ3りcま-しbた0が6、482時dに9洗7面8をeしf、2そ'の,後6人の王宮の神官や同志社の安藤氏などが来訪しました。その後、大阪支店長の野口氏や神戸・名古屋支店長などの諸氏と共に嵐山に着き、三軒屋で昼食を取り、船を浮かべて大江川を遡りました。船の中では囲碁をするなど楽しく過ごしました。嵐山の緑葉は少し少なくて繁茂していませんでしたが、両岸には山花が咲き乱れて風景がよかったです。午後4時すぎに玉川楼に帰宿し、その後尾崎司法大臣の秘書黒田氏や林和太郎氏（桂氏の父）が来て話をしました。午前10時から各支店主任と共に嵐山に着き、三軒屋で昼食を取り、大江川に船を浮かべて遡りました。午後4時に帰宿し、その後玉川楼で晩酌会を開き、種々の余興もありました。夜10時すぎに散会しましたが、中井三郎兵衛氏も来てくれました。',

June 14th was sunny with no wind and it was hot. I had a slight cold in the morning, but washed my face at 8 o’clock, after which officials from the Rokusonno Shrine and Mr. Ando from Doshisha visited. Then, together with Osaka branch manager Mr. Noguchi, Kobe and Nagoya branch managers and others, we went to Arashiyama, had lunch at Sangenya, and rowed a boat up the Oi River. We enjoyed playing Go on the boat. The green foliage at Arashiyama was slightly sparse, but mountain flowers bloomed in profusion on both banks, making for beautiful scenery. We returned to Tamagawa-ro after 4 PM. Secretary Kuroda of Justice Minister Ozaki and Mr. Hayashi Wataro (father of Mr. Katsura) came to visit. From 10 AM, together with branch managers, we went to Arashiyama, had lunch at Sangenya, and rowed a boat on the Oi River. We returned at 4 PM, then held a dinner party at Tamagawa-ro with various entertainments. The gathering ended at 10 PM, with Mr. Nakai Sarobei also attending.

When changing the model to llama3.1:latest (8.0B), the result was returned in about 6 seconds.

(The 8B model output was less natural in Japanese compared to the 70B model.)

While it takes more time, the llama3.1:70b (70.6B) model seems to produce more natural Japanese.

Summary

I ran a local LLM using mdx.jp’s 1GPU pack and Ollama.

I hope this serves as a helpful reference for using local LLMs.