{"id":21688,"date":"2026-08-15T19:53:55","date_gmt":"2026-08-15T17:53:55","guid":{"rendered":"https:\/\/www.juust.org\/?p=21688"},"modified":"2026-08-15T22:40:38","modified_gmt":"2026-08-15T20:40:38","slug":"building-a-contextual-bandit-ai-agent-in-typescript","status":"publish","type":"post","link":"https:\/\/www.juust.org\/index.php\/building-a-contextual-bandit-ai-agent-in-typescript\/2026\/08\/","title":{"rendered":"Building a Contextual Bandit AI Agent in TypeScript"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Most AI <a href=\"https:\/\/www.juust.org\/index.php\/tag\/agent\/\" target=\"_blank\" rel=\"noreferrer noopener\">agent<\/a> workflows use static rules: &#8220;use the balanced profile&#8221; or &#8220;always review before publishing.&#8221; That works fine until the tradeoffs shift \u2014 when latency matters more than cost, or when quality suddenly justifies a deeper research pass. A contextual bandit replaces those static rules with a learned routing policy that adapts to your objective.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This guide builds on the multi-agent article workflow from the <a href=\"https:\/\/www.juust.org\/index.php\/data-based-learning-ts-agent-in-langgraph\/2026\/08\/\" target=\"_blank\" rel=\"noreferrer noopener\">LangGraph Database Agent TypeScript example<\/a>, replacing the fixed agent profile with a contextual bandit that learns which path through the workflow delivers the best results for your current goal \u2014 whether that&#8217;s budget, speed, quality, or a balanced tradeoff.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">1. What Is a Contextual Bandit?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A contextual bandit is a reinforcement learning algorithm that learns to pick the best action from a set of discrete choices. Unlike A\/B testing (which treats every choice equally forever), a bandit actively explores unknown options while exploiting known good ones. Unlike full reinforcement learning (which models sequences of decisions across time), a bandit treats each decision independently.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In plain terms: the bandit has a set of &#8220;arms&#8221; it can pull \u2014 think slot machines in a casino. Each arm has an unknown reward distribution. The bandit&#8217;s job is to maximize total reward by trying arms it hasn&#8217;t tried much (exploration) while favoring arms that have paid off well (exploitation). Add &#8220;context&#8221; \u2014 information about the current situation, like remaining budget or quality target \u2014 and you have a contextual bandit that adapts its choices to the circumstance.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this project, the arms are edges in a workflow graph: &#8220;start \u2192 research&#8221; vs &#8220;start \u2192 research-lite&#8221;, &#8220;writer \u2192 review&#8221; vs &#8220;writer \u2192 publish&#8221;, and so on. Each edge carries a utility score based on quality gain, cost, latency, retry risk, and success probability \u2014 weighted by your chosen objective.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">2. Project Overview<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The repository is on GitHub: <a href=\"https:\/\/github.com\/juustesout\/contextual-bandit-ai-agent-typescript\" target=\"_blank\" rel=\"noreferrer noopener\">github.com\/juustesout\/contextual-bandit-ai-agent-typescript<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The workflow produces blog articles through five stages \u2014 research, writing, review, publishing \u2014 but instead of picking one static profile (budget\/balanced\/quality\/speed) at the start, the contextual bandit re-evaluates at every decision point. It chooses the next node based on historical performance stored in PostgreSQL, using either the epsilon-greedy or UCB1 strategy.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">3. Project Setup<\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>git clone https:\/\/github.com\/juustesout\/contextual-bandit-ai-agent-typescript.git tsbandit\ncd tsbandit\nnpm install<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The dependencies are minimal: <code>pg<\/code> for PostgreSQL, <code>typescript<\/code> and <code>@types\/node<\/code> for compilation. No LangChain, no LangGraph, no Zod \u2014 just plain TypeScript and a raw <code>fetch<\/code>-based MiniAgent class.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">4. Environment Variables<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Create a <code>.env<\/code> file in the project root:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>OPENAI_API_KEY=sk-...\nOPENAI_MODEL=gpt-4o-mini\nDATAFORSEO_API_KEY=base64-of-login:password\nDATAFORSEO_BASE_URL=https:\/\/api.dataforseo.com\nWP_BASE_URL=https:\/\/yourblog.com\nWP_USERNAME=admin\nWP_APPLICATION_PASSWORD=xxxx\nJINA_API_KEY=jina_...\nDESKTOP_DB_HOST=host.docker.internal\nDESKTOP_DB_PORT=5433\nDESKTOP_DB_NAME=xxxx\nDESKTOP_DB_USER=xxxx\nDESKTOP_DB_PASSWORD=...<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">5. The Workflow Graph<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The contextual bandit models the workflow as a directed graph. Each node is an execution step, and each edge is a possible transition. The bandit chooses at every branching point:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>start \u2192 &#91;research, research-lite]\nresearch \u2192 &#91;writer, fallback\/retry]\nresearch-lite \u2192 &#91;writer]\nwriter \u2192 &#91;review]\nreview \u2192 &#91;publish, fallback\/retry]\nfallback\/retry \u2192 &#91;writer]<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The graph is defined in <code>src\/bandit\/types.ts<\/code>. The entry decision (<code>start<\/code>) is a virtual node so the first edge is scored exactly like every other edge.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">6. The Utility Function<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Every edge is scored with a weighted utility function:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>utility = \u03b4 \u00d7 Q\u2099\u2092\u1d63\u2098  \u2212  \u03b1 \u00d7 C\u2099\u2092\u1d63\u2098  \u2212  \u03b2 \u00d7 L\u2099\u2092\u1d63\u2098  \u2212  \u03b3 \u00d7 retryRisk  +  \u03b4 \u00d7 successProb<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Each raw value is normalized to a 0\u20131 scale so the weights are objective-relative, not dominated by absolute magnitudes:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Q\u2099\u2092\u1d63\u2098<\/strong> = qualityGain \/ 100 (review score, 0\u2013100)<\/li>\n\n\n\n<li><strong>C\u2099\u2092\u1d63\u2098<\/strong> = costUsd \/ 0.05 (max ~$0.05 per edge)<\/li>\n\n\n\n<li><strong>L\u2099\u2092\u1d63\u2098<\/strong> = latencyMs \/ 120,000 (120s timeout)<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The weights per objective live in <code>src\/bandit\/utility.ts<\/code>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>const DEFAULT_UTILITY_WEIGHTS = {\n  budget:   { alpha: 0.5, beta: 0.3, gamma: 0.2, delta: 0.6 },\n  balanced: { alpha: 0.2, beta: 0.15, gamma: 0.2, delta: 0.8 },\n  quality:  { alpha: 0.05, beta: 0.05, gamma: 0.2, delta: 1.0 },\n  speed:    { alpha: 0.1, beta: 0.4, gamma: 0.15, delta: 0.6 },\n};<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Quality mode pushes \u03b4 (delta) to 1.0 and cuts \u03b1\/\u03b2 to near zero \u2014 quality dominates the decision. Budget mode raises \u03b1 to penalize cost. Speed mode raises \u03b2 to penalize latency.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">7. Exploration Strategies<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The contextual bandit supports two strategies, both in <code>src\/bandit\/strategies.ts<\/code>:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Epsilon-Greedy<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">With probability \u03b5, pick a random arm. Otherwise, pick the arm with the highest mean reward. Simple, effective, and the \u03b5 parameter tunes the exploration\/exploitation tradeoff directly. Default \u03b5 = 0.1.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">UCB1 (Upper Confidence Bound)<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Pick the arm that maximizes <code>meanReward + \u221a(2 \u00d7 ln(totalPulls) \/ armPulls)<\/code>. This automatically balances exploration and exploitation: arms with few pulls get a confidence bonus, while well-tested arms are chosen by their proven mean. No separate \u03b5 parameter needed.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Both strategies enforce a cold-start phase: arms with fewer than <code>minSamples<\/code> observations are explored uniformly before exploitation kicks in. This ensures the bandit has baseline data for every option before it starts optimizing.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">8. Learning from Quality: Backpropagation<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">An earlier version of the project had a fundamental flaw: the quality gain per edge was calculated as a hardcoded delta between node scores (e.g., writer.score \u2212 research.score = 82 \u2212 80 = 2). This was constant and identical for both research paths, so the bandit learned nothing about the actual article quality and defaulted to the cheapest route.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The fix: after the reviewer scores the final article with a <code>quality_score<\/code> (0\u2013100) based on depth, source variety, and factual substantiation, that score is backpropagated to every edge that contributed to the run. Every observation in the run gets the same quality credit \u2014 the bandit learns that a run that produced a high-quality article (usually from deeper research) rewards all its edges equally. This is implemented in the <code>observe()<\/code> function and the post-review backpropagation step in <code>src\/workflow.ts<\/code>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">9. PostgreSQL Persistence<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The bandit stores per-edge statistics in a PostgreSQL table called <code>edge_stats<\/code>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>CREATE TABLE edge_stats (\n  source_node      TEXT NOT NULL,\n  target_node      TEXT NOT NULL,\n  objective        TEXT NOT NULL,\n  times_selected   INTEGER NOT NULL DEFAULT 0,\n  cumulative_reward DOUBLE PRECISION NOT NULL DEFAULT 0,\n  avg_reward       DOUBLE PRECISION NOT NULL DEFAULT 0,\n  sum_quality_gain DOUBLE PRECISION NOT NULL DEFAULT 0,\n  sum_cost_usd     DOUBLE PRECISION NOT NULL DEFAULT 0,\n  sum_latency_ms   DOUBLE PRECISION NOT NULL DEFAULT 0,\n  success_count    INTEGER NOT NULL DEFAULT 0,\n  retry_count      INTEGER NOT NULL DEFAULT 0,\n  rejection_count  INTEGER NOT NULL DEFAULT 0,\n  last_updated     TIMESTAMPTZ NOT NULL DEFAULT NOW(),\n  PRIMARY KEY (source_node, target_node, objective)\n);<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Each run upserts a row per traversed edge, including failed and retried ones. The <code>aggregateEdgeStatsFromRows()<\/code> function in <code>src\/database.ts<\/code> turns these raw rows into the per-edge statistics the bandit consumes for cold-starting the next run.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">10. Running the Workflow<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Profile mode (static agent profile \u2014 no bandit):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>npm run build &amp;&amp; node dist\/demo.js \"AI sales agents for SMBs\"<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Bandit mode (learned routing):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>npm run build &amp;&amp; node dist\/demo.js \"AI sales agents\" --bandit --objective quality --strategy ucb1<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">11. Interpreting the Output<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A bandit run prints an edge-level decision log:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;bandit] start -&gt; research-lite | EXPLORE | expectedUtility=0.0000 | Cold start: all 2 arm(s) below 5 samples\n&#91;bandit] writer -&gt; review | EXPLORE | expectedUtility=0.0000 | Cold start: all 1 arm(s) below 5 samples\n&#91;bandit] review -&gt; publish | EXPLORE | expectedUtility=0.0000 | Cold start: all 2 arm(s) below 5 samples<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The routing object shows the full route, per-decision details, and the run reward:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\"routing\": {\n  \"mode\": \"bandit\",\n  \"objective\": \"quality\",\n  \"strategy\": \"ucb1\",\n  \"route\": &#91;\"research-lite\", \"writer\", \"review\"],\n  \"runReward\": 2.07,\n  \"edgeRewards\": {\n    \"start-&gt;research-lite\": 27.78,\n    \"research-lite-&gt;writer\": 56.75,\n    \"writer-&gt;review\": 16.56\n  }\n}<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">12. Training the Bandit<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The included <code>train-bandit.js<\/code> script runs the workflow N times and generates an HTML report with Chart.js visualizations:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>node train-bandit.js 15 \"AI sales agents\" quality<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This runs 15 workflows, collects per-edge statistics after each run, and produces three charts:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Average Reward per Edge<\/strong> \u2014 convergence of the utility estimate over time<\/li>\n\n\n\n<li><strong>Cumulative Reward<\/strong> \u2014 which edges accumulate the most value<\/li>\n\n\n\n<li><strong>Arm Selection Frequency<\/strong> \u2014 how often each edge was chosen after cold start<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">A typical 15-run training session completes in about 8\u201310 minutes and costs roughly $0.08 in API fees. The report is a self-contained HTML file \u2014 open it in any browser.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">13. Architecture Notes<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Discrete arms, not continuous<\/strong> \u2014 the bandit works on discrete edges (research vs lite, review vs publish). Continuous parameters (depth, scrape limit) would need Gaussian Process bandits or policy gradients, which overcomplicate the architecture for an agent workflow.<\/li>\n\n\n\n<li><strong>Objective-relative weights<\/strong> \u2014 the utility function normalizes all raw values to 0\u20131 before applying weights. This prevents latency (~90,000ms) from dominating quality (~80 points) purely by scale.<\/li>\n\n\n\n<li><strong>Review is mandatory<\/strong> \u2014 the <code>writer \u2192 publish<\/code> shortcut was removed from the graph. Every article must pass through review, ensuring a quality gate before publication.<\/li>\n\n\n\n<li><strong>Failures are learning signals<\/strong> \u2014 retried and rejected edges are recorded with negative rewards. The bandit learns to avoid unreliable paths, not just to favor successful ones.<\/li>\n\n\n\n<li><strong>Quality is backpropagated<\/strong> \u2014 the reviewer&#8217;s <code>quality_score<\/code> is folded back into every edge in the run (except retry paths). Every edge that contributed to a high-quality article shares the credit, so the bandit learns which routes produce the best final output.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">14. Going Further<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">This architecture is a foundation for adaptive agent routing. Natural extensions include:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Continuous research depth<\/strong> \u2014 add 3\u20134 discrete depth arms (lite\/standard\/deep) instead of the binary research vs lite choice.<\/li>\n\n\n\n<li><strong>Cross-objective transfer learning<\/strong> \u2014 share edge statistics between objectives with a Bayesian prior so quality runs benefit from data collected under balanced runs.<\/li>\n\n\n\n<li><strong>Online retraining<\/strong> \u2014 run the bandit in a background cron job that periodically retrains on the accumulated edge_stats data and adjusts weights.<\/li>\n\n\n\n<li><strong>Human-in-the-loop review<\/strong> \u2014 replace or augment the LLM reviewer with a human rating, feeding that score back through the same backpropagation mechanism.<\/li>\n\n\n\n<li><strong>Multi-objective Pareto frontier<\/strong> \u2014 instead of a single objective, track the Pareto frontier of quality vs cost vs latency and let the operator choose the operating point.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">15. Get the Code<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The full source code is on GitHub: <a href=\"https:\/\/github.com\/juustesout\/contextual-bandit-ai-agent-typescript\" target=\"_blank\" rel=\"noreferrer noopener\">github.com\/juustesout\/contextual-bandit-ai-agent-typescript<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Clone it, configure your API keys, point it at a PostgreSQL instance, and run <code>npm run build &amp;&amp; node dist\/demo.js \"your topic\" --bandit --objective quality<\/code>. The training script will show you what the bandit learns over 15 runs.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Building a Contextual Bandit in a TypeScript Agent &#8211; with repo<\/p>\n","protected":false},"author":5796,"featured_media":21634,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_sitemap_exclude":false,"_sitemap_priority":"","_sitemap_frequency":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"set","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[484,543,305,26],"tags":[483,546,545],"class_list":["post-21688","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","category-agents","category-programming","category-trends","tag-ai","tag-contextual-bandit","tag-pareto"],"_links":{"self":[{"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/posts\/21688","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/users\/5796"}],"replies":[{"embeddable":true,"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/comments?post=21688"}],"version-history":[{"count":3,"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/posts\/21688\/revisions"}],"predecessor-version":[{"id":21724,"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/posts\/21688\/revisions\/21724"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/media\/21634"}],"wp:attachment":[{"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/media?parent=21688"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/categories?post=21688"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.juust.org\/index.php\/wp-json\/wp\/v2\/tags?post=21688"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}