In this article, you will learn how to serve a Scikit-LLM zero-shot text classification pipeline, powered by a local Ollama model, as a REST API with FastAPI. Topics we will cover include: How to configure Scikit-LLM to use a local Ollama model instead of a paid cloud API. How to build and serialize a zero-shot classifier using the familiar scikit-learn fit/predict syntax. How to serve the classifier through an asynchronous FastAPI endpoint and test it from the interactive docs page. Introduction It looks like Scikit-LLM is here to stay. This innovative Python framework pushes the boundaries of machine learning and LLMs by treating them as two sides of the same coin, enabling scikit-learn-based pipelines fueled by powerful pre-trained LLMs adapted to perform specific tasks — for instance, building a smart zero-shot classification engine. All of this sounds enthralling until you try to put it into production and instantly get chained to costly cloud APIs, latency spikes, and so on. This article covers the first crucial steps toward deploying a deep semantic reasoning pipeline fueled by an LLM and combined with the reliability of standardized machine learning syntax — enter the .fit() and .predict() methods — as well as predictable outputs like those found in traditional machine learning pipelines. We will put together an open-source stack composed of Scikit-LLM, FastAPI, and Ollama, teaching you how to wire up a local, self-hosted LLM that acts as a deterministic text labeler: all wrapped in an asynchronous, blazing-fast REST API, thereby bypassing cloud costs and commercial lock-in. Setup and Initial Configurations For this practical demonstration, I have used Visual Studio Code with a virtual environment using Python 3.11. I have also installed some basic libraries like pandas and scikit-learn in the virtual environment. In addition, you’ll need to install the following specialized libraries to make the code work: pip install scikit-llm fastapi uvicorn joblib pydantic You’ll also need to install Ollama and pull (locally download) llama3. This article explains the process to install Ollama in more detail, but chances are you won’t need it if you just visit the Ollama website: it’s pretty simple to install, I promise! Building, Deploying, and Testing the Pipeline Once everything is correctly installed and you have resolved any possible issues caused by version compatibility with your Python interpreter, let’s move on to the core of the process. We’ll create a new project folder in our IDE and create two Python files inside it, which will contain the code: name them train.py and app.py. Let’s start with train.py. In this Python file, we will add the logic to train and serialize the Scikit-LLM pipeline. To keep things simple but scalable, we will just invoke the Ollama model and attach it to a one-stage pipeline consisting of a zero-shot classifier — in simple terms, a pre-trained LLM turned into a classifier by giving it the set of possible categories, or labels, from an example dataset of texts, without actual model training. After that, we serialize the pipeline: once you run the code below, a file named scikit_llm_model.joblib containing the serialized pipeline will be created in your project folder. Here’s the code to do this: 123456789101112131415161718 import joblibfrom skllm.config import SKLLMConfigfrom skllm.models.gpt.classification.zero_shot import ZeroShotGPTClassifier# 1. Routing Scikit-LLM to local Ollama instead of cloud APIs (usually paid)SKLLMConfig.set_gpt_url("http://localhost:11434/v1/")SKLLMConfig.set_openai_key("ollama-dummy")X = ["The battery drains too fast", "I love the new camera", "Screen is completely cracked"]y = ["Battery Issue", "Positive Feedback", "Hardware Damage"]# 2. Appending the specific Ollama model name after custom_url::clf = ZeroShotGPTClassifier(model="custom_url::llama3")clf.fit(X, y)# 3. Serializing the estimatorjoblib.dump(clf, "scikit_llm_model.joblib")print("Pipeline serialized successfully.") The next step is to add the code for the app.py file. Run this code after generating the serialized pipeline. This file is needed to: Reset and re-apply the Scikit-LLM configuration needed for the serving environment. Create a FastAPI app instance and define the predict() endpoint with the necessary logic to enable inference by the model once deployed. Note that the service can take multiple input texts in a single call, thus returning multiple class predictions in one shot. Start the server using uvicorn, which the final lines of code take care of. 123456789101112131415161718192021222324252627282930313233343536373839404142 from contextlib import asynccontextmanagerimport joblibimport asynciofrom fastapi import FastAPI, HTTPExceptionfrom pydantic import BaseModelfrom skllm.config import SKLLMConfig# Applying configuration for the serving environmentSKLLMConfig.set_gpt_url("http://localhost:11434/v1/")SKLLMConfig.set_openai_key("ollama-dummy")ml_models = {}@asynccontextmanagerasync def lifespan(app: FastAPI): # Loading model into memory on startup ml_models["classifier"] = joblib.load("scikit_llm_model.joblib") yield ml_models.clear()app = FastAPI(lifespan=lifespan)class InferenceRequest(BaseModel): texts: list[str]@app.post("/predict")async def predict(request: InferenceRequest): clf = ml_models["classifier"] try: # Offload blocking Scikit-LLM I/O to a background thread predictions = await asyncio.to_thread(clf.predict, request.texts) return {"predictions": predictions.tolist()} except Exception as e: raise HTTPException(status_code=500, detail=str(e))if __name__ == "__main__": import uvicorn # Starts the FastAPI server when you run 'python app.py' uvicorn.run(app, host="127.0.0.1", port=8000) Once you run this code in your IDE, if everything goes smoothly, you should see the following messages in the output console: INFO: Started server process [9351]INFO: Waiting for application startup.INFO: Application startup complete.INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit) Now your pipeline is accessible (well, locally accessible, I mean), and you can open the FastAPI interface by browsing to http://127.0.0.1:8000/docs. To test the app, click the “Try it out” button inside the expandable “/predict” block, and edit the “Request body” text box to paste the following input examples or any of your own: { "texts": [ "My phone dies by noon even if I don't use it.", "I dropped it on the sidewalk and the glass shattered.", "This is the best device I have ever purchased!" ]} Then click “Execute”, and your pipeline’s LLM will start doing its magic. In a few seconds, the predictions will pop up: { "predictions": [ "Battery Issue", "Hardware Damage", "Positive Feedback" ]} Final Remarks Congratulations if you made it here! Just be aware of a few notes about what we have just done together: The classifier we built and deployed relies on the label set y we passed to it in the fit() method early on. We didn’t update the LLM’s weights (this process would have taken considerably longer!). Instead, the classifier stored the specific closed-domain set of categories to choose from when performing inference on texts, just as if we had written a multiple-choice question in a prompt. For this reason, when turning a pre-trained model into a zero-shot classifier, we are restricting its “view of the world”, forcing it to map any input (no matter what) to exactly one of the labels in the label set. In exchange for this closed-world view, we add a layer of reliability and predictability to an LLM that already had a profound level of knowledge. We built an extremely simple, one-stage pipeline, but thanks to Scikit-LLM’s compatibility with scikit-learn objects and syntax, we could have created a larger pipeline with additional preprocessing steps, similar to the example in this related article. No comments yet.
Serving Scikit-LLM Pipelines as REST APIs with FastAPI
Full Article
Original Source
Read the full article at Machinelearningmastery →KhanList aggregates and links to publicly available news content. We do not host full articles from third-party sources. Always verify important information with original sources.