{"id":10629,"date":"2023-12-09T11:04:40","date_gmt":"2023-12-09T09:04:40","guid":{"rendered":"https:\/\/www.juust.org\/?p=10629"},"modified":"2023-12-09T13:05:39","modified_gmt":"2023-12-09T11:05:39","slug":"lm-studio","status":"publish","type":"post","link":"https:\/\/www.juust.org\/index.php\/lm-studio\/2023\/12\/","title":{"rendered":"LM Studio met Next.js : je eigen lokale AI"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Ik had gisteren als doel : AI. Ik zat een video te kijken over &#8216;5 portfolio projecten&#8217;, Leuke ideeen voor kleine demo-projectjes. Van die dingen die je in een dag of twee dagen in het weekend even bouwt. Arduino-botjes en Python desktop task automation. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ik zat zelf eerder aan AI te denken. Onze vrolijke ijverige nerds hebben de LLM bedacht, &#8220;large language model&#8221;. De  programmeurs wereld is er vol van, &#8220;ChatGPT&#8221;. Vooral ook omdat het een API heeft. Dan kun je zelf ook AI programmaatjes maken. Let me at it, let me at it&#8230; maar ik heb geen credit card (de meeste AI API&#8217;s zijn Amerikaanse bedrijven en die werken met Stripe\/credit cards). En ik heb geen zin om 240 per jaar te betalen om voor mijn eigen lol met een API te spelen. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ik zocht een gratis AI met API.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Dus ik ging donderdagavond eens rondkijken, en vrijdagochtends nog een tijdje. Sites als HuggingFace kende ik al,  HuggingFace is de centrale repo voor open source LLM&#8217;s. Maar ik kon er op dat moment niks mee. Ook omdat ik op dat moment de tijd er niet voor nam. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Nu zat ik lekker rustig met mijn kopje koffie.  <\/p>\n\n\n\n<h3 class=\"wp-block-heading\">LM Studio<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Ik kwam uiteindelijk een leuke tool tegen, <a href=\"https:\/\/lmstudio.ai\/\" data-type=\"link\" data-id=\"https:\/\/lmstudio.ai\/\" target=\"_blank\" rel=\"noopener\">LM Studio<\/a>. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Dat is een gratis utility. Daarmee kun je elk LLM (large language model) van HuggingFace downloaden. Je hebt keuze uit tientallen modellen. Je kunt de modellen downloaden, mounten, en er mee chatten. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ik koos een &#8220;Llama 2&#8221; model, en pikte het kleinste model, 2.9 GB. Dat laadt vrij snel.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/www.juust.org\/wp-content\/uploads\/2023\/12\/selectllm.jpg\"><img fetchpriority=\"high\" decoding=\"async\" width=\"800\" height=\"206\" src=\"https:\/\/www.juust.org\/wp-content\/uploads\/2023\/12\/selectllm.jpg\" alt=\"\" class=\"wp-image-10631\" srcset=\"https:\/\/www.juust.org\/wp-content\/uploads\/2023\/12\/selectllm.jpg 800w, https:\/\/www.juust.org\/wp-content\/uploads\/2023\/12\/selectllm-300x77.jpg 300w, https:\/\/www.juust.org\/wp-content\/uploads\/2023\/12\/selectllm-768x198.jpg 768w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/><\/a><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Chat<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Even je model mounten bovenin het scherm, en dan kun je gaan chatten met je model : (hint : je moet linksboven in de interface wel even klikken op &#8216;new chat&#8217;, anders kun je niet invoeren in de prompt-regel).<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/www.juust.org\/wp-content\/uploads\/2023\/12\/chatlocal.png\"><img decoding=\"async\" width=\"600\" height=\"223\" src=\"https:\/\/www.juust.org\/wp-content\/uploads\/2023\/12\/chatlocal.png\" alt=\"\" class=\"wp-image-10632\" srcset=\"https:\/\/www.juust.org\/wp-content\/uploads\/2023\/12\/chatlocal.png 600w, https:\/\/www.juust.org\/wp-content\/uploads\/2023\/12\/chatlocal-300x112.png 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/a><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Sneeuwwitje als tekst kent &#8216;ie ! Hij weet ook wat het goeie antwoord is. <\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Lokale Http API<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Interessanter voor mij : je kunt ook een lokale http server opstarten. Die een beperkte OpenAI stijl API biedt. Twee endpoints in feite : completion en chat\/completion, dat is nog heel beperkt. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Maar dan kun je lokaal er al tegen aan programmeren. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In mijn geval met Next.js.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/www.juust.org\/wp-content\/uploads\/2023\/12\/localserver.jpg\"><img decoding=\"async\" width=\"700\" height=\"415\" src=\"https:\/\/www.juust.org\/wp-content\/uploads\/2023\/12\/localserver.jpg\" alt=\"\" class=\"wp-image-10630\" srcset=\"https:\/\/www.juust.org\/wp-content\/uploads\/2023\/12\/localserver.jpg 700w, https:\/\/www.juust.org\/wp-content\/uploads\/2023\/12\/localserver-300x178.jpg 300w\" sizes=\"(max-width: 700px) 100vw, 700px\" \/><\/a><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Ik heb CORS hier even aangezet (staat standaard uit) omdat ik even geen zin had om dat in te regelen. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Verder kun je in Next.js gewoon de OpenAi JS library gebruiken. Met in je OpenAI instance als config : apiKey: &#8216;NULL&#8217;, BaseURL : (je lokale http server url) + &#8216;\/v1\/&#8217;. Dan gebruikt de OpenAI library je lokale LLM met de lokale http server als bron API.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>\/app\/utils\/openai.defs.ts<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import OpenAI from \"openai\";\n\nexport type ChatCompletion = OpenAI.Chat.Completions.ChatCompletion;\nexport type ChatCompletionOptions =\n  OpenAI.Chat.Completions.ChatCompletionCreateParams;\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>\/app\/utils\/openai.ts<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import OpenAI from \"openai\";\n\nimport type { ChatCompletion } from \".\/openai.defs\";\n\nconst openai = new OpenAI({\n  baseURL: \"http:\/\/localhost:1234\/v1\/\",\n  apiKey: \"NULL\",\n  dangerouslyAllowBrowser: true,\n});\n\nexport default async function getChat(): Promise&lt;ChatCompletion&gt; {\n  return await openai.chat.completions.create({\n    messages: &#91;\n      {\n        role: \"user\",\n        content: \"Say this is a test\",\n      },\n    ],\n    model: \"gpt-3.5-turbo\",\n  });\n}\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>\/app\/page.tsx<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\"use client\";\n\nimport type { ChatCompletion } from \"@\/app\/utils\/openai.defs\";\n\nimport { useEffect, useState } from \"react\";\n\nimport getChat from \"@\/app\/utils\/openai\";\n\nexport default function Home() {\n  const &#91;comp, setComp] = useState&lt;ChatCompletion | null&gt;(null);\n\n  const fetchData = async () =&gt; {\n    try {\n      const mycomp: ChatCompletion = await getChat();\n      setComp(mycomp);\n    } catch (error) {\n      console.error(\"Error fetching data:\", error);\n    }\n  };\n\n  useEffect(() =&gt; {\n    fetchData();\n  }, &#91;]); \n\n  return (\n    &lt;main className=\"flex min-h-screen flex-col items-center justify-between p-24\"&gt;\n      &lt;div&gt;{comp?.choices&#91;0].message.content}&lt;\/div&gt;\n    &lt;\/main&gt;\n  );\n}\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Dit is Next 14.0.3 met App router, TypeScript en openai 4.20.1. Sorry voor mijn zeer beperkte Next.js skills, ik ben er net mee begonnen :) Dit moet je even zo coden omdat OpenAI JS een Node module is, en niet in een client component gebruikt kan worden in Next. Je moet de type definities (voor TypeScript) even in een aparte file exporteren.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Met de prompt &#8220;Say this is a test ?&#8221; krijg je dit eruit : <\/p>\n\n\n\n<pre class=\"wp-block-code has-ast-global-color-6-background-color has-background\"><code>Sure, I'm happy to help you with your test! Is there anything specific you would like me to assist you with? Please let me know how I can help. Also, please note that I'm just an AI and do not have the ability to give you a passing grade or provide any guarantees of success. My purpose is to assist and provide helpful responses to your questions to the best of my abilities.<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Heel welbespraakt ! Duurt wel even, wees geduldig, het kost 10 seconden om het antwoord aan te maken op mijn machine. Anderen hebben weer veel snellere machines).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Nogmaals, sorry voor de clumsy code :) maar ik vind dingen als dit gewoon ontzettend leuk om even mee te spelen. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Note: naast LM Studio zijn er meer lokale AI apps, die ongeveer hetzelfde doen. Op <a href=\"https:\/\/dev.to\/raphiki\/series\/24682\" data-type=\"link\" data-id=\"https:\/\/dev.to\/raphiki\/series\/24682\" target=\"_blank\" rel=\"noopener\">dev.to<\/a> hebben ze een kort overzicht van de andere voornaamste kandidaten (GTP4All en LocalAI) .<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>LM studio is een tool om lokaal een LLM large language model te installeren, met een lokale http API<\/p>\n","protected":false},"author":5796,"featured_media":10581,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_sitemap_exclude":false,"_sitemap_priority":"","_sitemap_frequency":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"set","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[305,11,26],"tags":[464],"class_list":["post-10629","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-programming","category-seo-tool","category-trends","tag-openai"],"_links":{"self":[{"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/posts\/10629","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/users\/5796"}],"replies":[{"embeddable":true,"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/comments?post=10629"}],"version-history":[{"count":5,"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/posts\/10629\/revisions"}],"predecessor-version":[{"id":10645,"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/posts\/10629\/revisions\/10645"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/media\/10581"}],"wp:attachment":[{"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/media?parent=10629"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/categories?post=10629"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/tags?post=10629"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}