pie

from inferlet import chat, model
from inferlet.eta import ForwardPass, Pipeline, StateGeometry, WorkingSet, intrinsics, reduce_argmax


async def main(input: dict) -> dict:
    tokens = chat.prefix() + model.encode(input.get("prompt", "The capital of France is"))
    max_tokens = int(input.get("max_tokens", 8))
    ws = WorkingSet()  # the sequence's state, whatever the model keeps
    pipe = Pipeline()

    done = 0  # tokens already in the working set
    for _ in range(max_tokens):
        fwd = ForwardPass()
        fwd.embed(tokens[done:])                        # the new tokens
        fwd.state([ws], StateGeometry.folding())        # appended, and final
        out = fwd.epilogue(lambda: reduce_argmax(intrinsics.output(0, "logits")))  # runs on the device
        pipe.submit(fwd)
        done = len(tokens)
        tokens.append(await out)
    pipe.close()

    return {"text": model.decode(tokens[-max_tokens:])}

epilogue is traced once and compiled into a GPU kernel; the host only sees the token it publishes. More in examples/: device loop-carried decode, speculative decoding, beam search, KV eviction and prefix sharing, constrained decoding, custom samplers, watermarking, and media generation.

Run & serve

Run an inferlet one-shot, or submit it to a running pie serve with pie-client:

pie run main.py -- --prompt "The capital of France is"
pie-client submit --path main.py -- --prompt "The capital of France is"

pie serve also speaks the OpenAI, Anthropic, and Gemini APIs on the same port as built-in inferlets:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="unused")
print(client.chat.completions.create(model="default", messages=[{"role": "user", "content": "Hi"}]).choices[0].message.content)

A program is short and its state need not be. A client can own the state, and run a new program on it each turn: nothing stays loaded between turns, a fork costs no device memory, and a turn resumes exactly where the last one stopped.

ws = await client.working_set()                    # a name; nothing exists yet
await (await client.run("chat", ws=ws, prompt="Hi")).result()
checkpoint = await ws.fork()                       # to come back to
await (await client.run("chat", ws=ws, prompt="Go on")).result()
await (await client.run("chat", ws=checkpoint, prompt="Instead...")).result()

The inferlet finds the handle in its input and gives it back:

async def main(input: dict) -> dict:
    ws = input["ws"]                     # a WorkingSet, moved in
    ...
    ws.give("ws")
    return {"text": text}

A handle is in one place at a time, so the client's next use waits for the program to give it back. It lives as long as the client's connection, on one worker, and the planner may move it off the device while it rests. If that worker is lost the connection closes and says so; a turn is never run again on what no longer exists. A client.matcher(json_schema=...) moves the same way, beside the working set it constrains.

Pictures and sound work the same: a program gives them to the client, which saves them in the format it wants, or moves them on to another program with no bytes crossing the client:

out = await client.frames()
await (await client.run("image-gen", out=out, prompt="a cat")).result()
await out.save("cat.png")
await (await client.run("caption", image=out)).result()

See examples/working-set-handles.

To embed in a process, use pie-server (Python) or @pie-project/server (Node and WebGPU browsers).

Build from source

cargo build --release -p pie --bin pie --features metal   # or: cuda, vulkan, wgpu, rocm, xla

Community & License

Discord · Issues · Discussions · Contributing

Apache License 2.0 (NOTICE).