LLMs for low-resource programming languages

What happened, and why I’m paying attention

OpenAI has published a piece on advancing coding in low-resource programming languages. You can read it at OpenAI, and there’s a page section addressable as sqlite-fortran.

For me, the message is simple: don’t wait for a perfect model for your niche language. Treat the model as a probabilistic assistant, then box it in with your own data, your own compiler, and your own tests. If you do that, you can start extracting safe value from legacy codebases while the frontier models keep improving.

Why low-resource programming languages matter now

Most teams I meet have one or more systems in a language the broader web barely speaks about. That gap is the opportunity. Your code, your domain terms, and your build tools are the ground truth; the model is just an accelerator around them.

The goal isn’t magical LLM code generation. The goal is repeatable, measurable assistance: drafting routines that compile, suggesting migrations, summarising unfamiliar modules, and writing tests—all under tight feedback from your toolchain.

A 14‑day plan to make this real

Day 1: Pick a narrow scope. Choose a single library, service, or batch job in your target language. Define success as “more green tests and fewer broken builds,” nothing grander.

Day 2–3: Build a small curated corpus. Collect 20–50 representative files from your codebase that show idioms, data structures, naming, and error handling. Add a handful of hand-written, well-commented exemplars that show your preferred style. Store them in a repo folder the agent can read, separate from production code.

Day 3–5: Autogenerate tests from behaviour descriptions. For each function or module, write short, human descriptions of expected behaviour and edge cases. From those, script generation of skeleton tests in your language’s test framework. Keep these as the canonical contract the agent must satisfy.

The deployment decision behind this is covered in Private LLM for Enterprise: When It Is Worth It.

Day 4–6: Put the compiler in the loop. Every agent action that proposes code must be compiled and executed against the tests. Compilation errors and failing assertions become the model’s next prompt context, not a reason to stop.

Day 5–7: Wire retrieval for your APIs and types. Create a lightweight index over your internal docs, headers, interfaces, and message schemas. At generation time, retrieve the top few relevant snippets and attach them to the prompt. This is where code RAG earns its keep: the model chooses from what exists, rather than invents.

Day 7–9: Add a thin agent loop. One tool proposes code changes; one tool compiles and runs tests; one tool reports diffs and errors; and a final step decides to retry, revert, or submit a patch for review. Keep the loop narrow and observable.

Day 9–11: Prompting before anything else. Start with system prompts that emphasise your style, safety rules, and test-first outcomes. If you have enough curated pairs, consider light fine-tuning later—but only after your harness proves consistent gains.

Day 10–12: Stand up an evaluation pass. Freeze a set of tasks with known-good outputs. Run nightly and on every agent change. Track compile success rate, test pass rate, and time-to-green as your primary signals.

Day 12–14: Harden and socialise. Add logging, diff review, and a safe deployment path. Start letting a few engineers use the agent for small tickets under supervision. Iterate the corpus, tests, and prompts daily based on failures.

Before scaling, it is worth having someone review your LLM or agent architecture against the controls it will need.

Make the tooling do the thinking: compiler-in-the-loop and evals

The pivot is simple: make your toolchain the arbiter of truth. The agent proposes; your compiler and tests decide. That makes failure informative, not dangerous, and it creates data you can use to improve prompts and tasks.

A minimal compile-and-test runner can be this small. Note the placeholders—use your real compiler, flags, and test runner, and return machine-readable results back to the agent.

sh#!/usr/bin/env sh
set -eu
SRC_FILE="$1"
BUILD_DIR=".agent_build"
BIN="$BUILD_DIR/bin"
mkdir -p "$BUILD_DIR"

# Compile step: replace $COMPILER, $FLAGS, and $OUT as appropriate
if ! $COMPILER $FLAGS -o "$BIN" "$SRC_FILE" 2>"$BUILD_DIR/compile.err"; then
  printf '{"status":"compile_error"}\n'
  exit 0
fi

# Test step: replace with your real test runner; must emit JSON
if ! ./run_tests.sh "$BIN" >"$BUILD_DIR/results.json" 2>"$BUILD_DIR/test.err"; then
  printf '{"status":"test_error"}\n'
  exit 0
fi

# Always print a single JSON object the agent can parse
cat "$BUILD_DIR/results.json"

undefined

json{
  "task_id": "mod-parse-001",
  "source_files": ["src/parser.mod"],
  "goal": "Extend the parser to support unary minus without breaking existing precedence rules.",
  "constraints": [
    "Do not change public function signatures",
    "Preserve existing whitespace handling"
  ],
  "tests": {
    "positive": ["tests/unary_minus_ok.t"],
    "regression": ["tests/precedence_regressions.t"]
  }
}

undefined

Operational wiring: retrieval, prompts, and reviews

Retrieval: index your own interfaces, error codes, and style guides. At generation time, fetch a few relevant snippets and pin them to the prompt along with the task JSON. Keep the index small and curated at first; drift here will leak noise into every step.

Prompts: insist on constraints up front. Ask for diffs, not full files. Ask the model to explain compile or test failures briefly before another attempt. That pre-commit narrative helps reviewers and becomes training data later.

Reviews: land only changes that compile and pass tests. Gate on your nightly evals too; if they regress, block merging further agent changes until you’ve fixed the root cause. This keeps enthusiasm from outrunning reliability.

What to do now

– Read the signal from OpenAI and decide which one slice of your stack would benefit most from constrained assistance first.

– Pick three engineers and two weeks. Commit to the plan above, including the small corpus, retrieval, compiler-in-the-loop, and nightly evals.

– Start with one happy-path task and one gnarly edge-case. Let the harness tell you where to invest next.

– Keep scope narrow. Resist feature creep until the graphs move in the right direction for a week.

– Only after that consider expanding the corpus or introducing fine-tuning. Your discipline—not the model—is what makes this sustainable.