All work

Selected work / Case study

Ollama Provider

An OpenAI-compatible gateway for local Ollama models, combining per-client access policies, request scheduling, streaming, usage records, and a responsive administration dashboard.

The problem

Let applications share local models with separate credentials and usage limits, while giving operators a clear way to review requests, failures, and model availability.

Engineering focus

Coordinate model-aware scheduling, per-key admission, and streaming cleanup in a shared Python gateway, with asynchronous usage records and a separate administrative interface.

Implementation overview

  1. Client application

    OpenAI-compatible requests from client applications.

  2. Client policies

    Authentication, model restrictions, limits, and per-key admission.

  3. Shared gateway

    Model-aware scheduling, proxy streaming, and idempotent cleanup.

  4. Ollama runtime

    Pretrained inference in the separate Ollama runtime.

  5. Usage and administration

    Asynchronous SQLite records and an admin application serving a static Next.js UI.

Conceptual component overview derived from inspected source, not an observed production deployment.

Implementation

  • Expose OpenAI-compatible model listing, chat, text-completion, and embedding routes; validate each request against model capabilities.
  • Manage client keys with expiry, model restrictions, request/concurrency limits, optional token quotas, priorities, and prompt-logging choices.
  • Group requests by model and release scheduler tickets and per-key slots when a response completes or its stream is interrupted.
  • Write usage records asynchronously to SQLite; retrieve prompt previews with cursor pagination and full details on demand.
  • Provide a React/Next.js dashboard for overview, client policies, usage, requests, prompts, and models, exported as static pages served by the admin FastAPI application.

Evidence & source

The supplied report documents mocked backend regression checks and frontend browser journeys. Screenshots show the built application with demo fixtures; the architecture figure is source-derived, not a production deployment observation.

Decisions & limitations

Screenshots show the actual UI with isolated demo data: metrics, hardware status, queues, model inventory, download progress, and answers are simulated. Live GPU throughput and production recovery were not tested in the supplied review. Scheduling assumes one gateway process; soft quotas can be exceeded by in-flight requests. Prompt logging needs operator privacy controls.

Technology stack

  • Python
  • FastAPI
  • Ollama
  • SQLite
  • React
  • Next.js
  • TypeScript
  • Tailwind CSS