Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
---
title: Catalog-model inference API
description: >-
Serve an OpenAI-compatible inference endpoint from your own application
domain, with a function that calls the model with Azion.AI.run.
Serve an OpenAI-compatible inference endpoint from your own Application
domain, with a Function that calls the model with Azion.AI.run.
meta_tags: 'reference architecture, catalog-model inference API, AI Inference, Azion.AI.run, functions, applications, OpenAI-compatible endpoint, inference endpoint'
namespace: docs_arch_ai_inference
permalink: /documentation/use-cases/build-and-run-ai-workloads/ai-inference-architecture/
Expand All @@ -12,7 +12,7 @@ import FrameBox from '@aziontech/webkit/frame-box'
import ItemList from '@aziontech/webkit/item-list'
import DocItem from '@aziontech/webkit/doc-item'

A client that calls a model needs one address to post to and one response shape to read. This design puts that address on an application you deploy: the application is the inference endpoint, and the model runs behind it. Use it when your team already serves an HTTP API and wants inference to be one more path on it. You own the domain, and no inference server stays running. The design implements the use case [Add AI features to existing applications](/en/documentation/use-cases/build-and-run-ai-workloads/add-ai-features-to-existing-applications/).
A client that calls a model needs one address to post to and one response shape to read. This design puts that address on an Application you deploy: the Application is the inference endpoint, and the model runs behind it. Use it when your team already serves an HTTP API and wants inference to be one more path on it. You own the domain, and no inference server stays running. The design implements the use case [Add AI features to existing applications](/en/documentation/use-cases/build-and-run-ai-workloads/add-ai-features-to-existing-applications/).

[How AI Inference works](/en/documentation/platform/ai-inference/how-it-works/) covers the hand-offs inside a single call. This page describes the shape you deploy around them.

Expand All @@ -26,10 +26,12 @@ Solid arrows carry the request, and dotted arrows carry the response. Each label
%%{init: {"layout": "elk", "elk": {"cycleBreakingStrategy": "MODEL_ORDER"}, "themeVariables": {"fontSize": "13px"}, "flowchart": {"nodeSpacing": 24, "rankSpacing": 36, "padding": 8, "wrappingWidth": 150, "useMaxWidth": true}, "sequence": {"width": 120, "actorMargin": 30, "useMaxWidth": true}}}%%
flowchart TD
Client(["OpenAI-compatible client"])
App["application"]
Fn["function with the prompt and the model id"]
WL["Workload"]
App["Application"]
Fn["Function"]
AI["AI Inference"]
Client -->|"1, 2. POST /v1/chat/completions"| App
Client -->|"1. POST /v1/chat/completions"| WL
WL -->|"2. request"| App
App -->|"3. Run Function"| Fn
Fn -->|"4. Azion.AI.run"| AI
AI -.->|"5. generated response"| Fn
Expand All @@ -39,29 +41,29 @@ flowchart TD

### Dataflow

1. A client posts an OpenAI-compatible request to `/v1/chat/completions` on the application domain.
2. The nearest data center receives the request and hands it to the application.
3. The application applies its policies, checks the authentication you configured, and calls the function.
4. The function preprocesses the input and calls the model with `Azion.AI.run`, which reaches AI Inference through the Azion Cells API.
5. AI Inference runs the model and returns the generated response to the function.
6. The function shapes that response and returns it to the client along the same path.
1. A client posts an OpenAI-compatible request to `/v1/chat/completions` on a domain of the Workload.
2. A data center near the client receives the request, and the Workload's deployment sets the Application that handles it.
3. The Application runs its rules, including any authentication check you add, and a rule runs the Function with the _Run Function_ behavior.
4. The Function builds the request body and calls the model with `Azion.AI.run`.
5. AI Inference runs the model and returns the generated response to the Function.
6. The Function shapes that response and returns it to the client along the same path.

The client knows nothing about Azion. It posts JSON to a URL and reads JSON back, as it would against any OpenAI-compatible service. Behind the domain, one execution does the rest, and the function holds the request open until the model responds. No arrow leaves for an inference host, because the function configures no base URL and names no host. It passes a model id, and Azion places the call.
The client knows nothing about Azion. It posts JSON to a URL and reads JSON back, as it would against any OpenAI-compatible service. Behind the domain, one execution does the rest, and the Function holds the request open until the model responds. No arrow leaves for an inference host, because the Function passes no base URL and names no host. It passes a model id, and Azion places the call.

---

## Components

- [Applications](/en/documentation/platform/applications/): receives the request on your domain and applies its policies before any of your code runs. It is what makes the endpoint yours, because the domain, the path, and the authentication in front of the function are all configured here. Azion issues no credential for a model call, so an endpoint with no check of its own answers anyone who finds it.
- [Workloads](/en/documentation/platform/workloads/): receives the request on its domains, the workload domain or a custom domain, and its deployment sets the Application. The address the client posts to belongs to the Workload.
- [Applications](/en/documentation/platform/applications/): runs its rules on the request before any of your code runs. The path the Function answers and any authentication in front of it are set in those rules. `Azion.AI.run` takes no credential, so an endpoint with no check of its own answers anyone who finds it.
- [Functions](/en/documentation/platform/functions/): holds the inference logic and issues the model call. Your code runs nowhere else in the design, so the prompt, the model id, the context retrieved from a store, and any external service the call needs all sit in one execution.
- [AI Inference](/en/documentation/platform/ai-inference/): runs the model, which is what removes the inference server from the design. You load no model and size no cluster, and the id from [AI models](/en/documentation/platform/ai-inference/models/) is the only address the function needs.
- [Orchestrator](/en/documentation/platform/orchestrator/): manages the requests for Azion Cells execution. `Azion.AI.run` reaches a model through the Azion Cells API, which is the part of Orchestrator this design depends on.
- **Azion's distributed infrastructure**: runs every part above. It is why the design has no region to choose. The data center nearest the client answers the request, and you select none of them.
- [Real-Time Metrics](/en/documentation/platform/real-time-metrics/): reports the traffic and performance of the application that serves the endpoint. A design with no inference server of your own still needs a view of what the endpoint carried.
- [AI Inference](/en/documentation/platform/ai-inference/): runs the model, which is what removes the inference server from the design. You load no model and size no cluster, and the id from [AI models](/en/documentation/platform/ai-inference/models/) is the only address the Function needs.
- **Azion's distributed infrastructure**: runs every part above. It is why the design has no region to choose. A data center near the client answers the request, and you select none of them.
- [Real-Time Metrics](/en/documentation/platform/real-time-metrics/): reports the traffic and performance of the Application that serves the endpoint. A design with no inference server of your own still needs a view of what the endpoint carried.
- [Real-Time Events](/en/documentation/platform/real-time-events/): tracks individual requests, raw data, and logs. When a response is wrong, the request that produced it is what you read here.

:::note
Azion Cells is an advanced feature that is part of Orchestrator. To activate it, contact Azion's sales team to verify the requirements and enable the product in your account.
On the Hobby and Pro plans, AI Inference is available upon request, subject to approval. For the access each plan gives, refer to [AI Inference limits](/en/documentation/platform/ai-inference/limits/).
:::

---
Expand All @@ -70,7 +72,7 @@ Azion Cells is an advanced feature that is part of Orchestrator. To activate it,

- [Add AI features to existing applications](/en/documentation/use-cases/build-and-run-ai-workloads/add-ai-features-to-existing-applications/) - builds this design end to end, with the summarization function and the API route.
- [AI Inference quickstart](/en/documentation/platform/ai-inference/quickstart/) - deploys the design and posts a first request to a model on the domain it assigns.
- [How to deploy the AI Inference Starter Kit template](/en/documentation/guides/application-development/frameworks/ai-inference-starter-kit/) - creates the application, the function, and the domain this page describes, in one deployment.
- [How to deploy the AI Inference Starter Kit template](/en/documentation/guides/application-development/frameworks/ai-inference-starter-kit/) - creates the Workload with its deployment, the Application, and the Function this page describes, in one deployment.
- [Model invocation](/en/documentation/platform/ai-inference/model-invocation/) - carries the request body both interfaces accept, and the `curl` form for the HTTP endpoint.

---
Expand All @@ -82,7 +84,7 @@ Azion Cells is an advanced feature that is part of Orchestrator. To activate it,
<DocItem title="How AI Inference works" href="/en/documentation/platform/ai-inference/how-it-works/">The invocation chain hand-off by hand-off, and where Azion's part ends.</DocItem>
<DocItem title="AI models" href="/en/documentation/platform/ai-inference/models/">The id to pass, and what each model accepts.</DocItem>
<DocItem title="AI Inference limits" href="/en/documentation/platform/ai-inference/limits/">The ceilings a model call runs against, and when Azion deprovisions a model.</DocItem>
<DocItem title="SQL Database" href="/en/documentation/platform/sql-database/">The store a function reads context from before it calls a model.</DocItem>
<DocItem title="SQL Database" href="/en/documentation/platform/sql-database/">The store a Function reads context from before it calls a model.</DocItem>
<DocItem title="Vector search" href="/en/documentation/platform/sql-database/vector-search/">The embedding queries that build the context a prompt carries.</DocItem>
</ItemList>
</FrameBox>
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ Definitions for the terms the [Functions](/en/documentation/platform/functions/)
| Args | The JSON object a [function instance](/en/documentation/platform/applications/functions-instances/) passes into the execution context of the function. An instance that sets Args replaces the default Args of the function as a whole. [How it works](/en/documentation/platform/functions/how-it-works/) shows which Args the function receives. |
| Azion Bundler | The open-source framework adapter that transforms a framework project into functions that run on Azion Runtime. [How it works](/en/documentation/platform/functions/how-it-works/) describes the four stages of the build, from framework detection to the deployment of the generated functions. |
| Azion Runtime | The environment that executes the handler of a function. [Azion Runtime](/en/documentation/devtools/runtime/) runs JavaScript built on Web standards. A function also calls the [Azion APIs](/en/documentation/devtools/runtime/api-reference/azion-runtime-globals/), which reach environment variables, stores, and AI Inference. |
| Cell | The isolation environment a function runs inside, based on V8 isolates. Each Cell is a lightweight execution context with no direct access to the underlying operating system, which is why a function cannot read OS properties, resolve DNS through native system calls, or open low-level sockets. [Node.js compatibility](/en/documentation/devtools/runtime/node/) lists what that excludes. |
| code editor | The editor in Azion Console where you write and edit the code of a function. The [code editor](/en/documentation/platform/functions/code-editor/) runs the Monaco editor from VS Code, and shows the output of the function through Preview deployment. |
| CPU time | The active computation a single invocation performs, capped at 2 seconds. It excludes time spent waiting on `fetch()`, so a function that waits on an origin stays far from this ceiling however long the request takes. A function that exceeds it is terminated. Compare wall-clock time, and refer to [Limits](/en/documentation/platform/functions/limits/). |
| environment variable | A value stored outside the code of a function and read with `Azion.env.get()` or `process.env`. In a deployed function, the `env` parameter of the handler is an empty object. [Environment variables](/en/documentation/platform/functions/environment-variables/) keep API keys, credentials, and access tokens out of the codebase. |
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: AI Inference API
description: >-
Call the Azion.AI.run binding inside a function to invoke an AI Inference
Call the Azion.AI.run binding inside a Function to invoke an AI Inference
model, and look up its parameters and return value.
meta_tags: 'AI Inference, Azion.AI.run, runtime binding, model invocation, functions, runtime api'
namespace: documentation_runtime_ai
Expand All @@ -12,30 +12,30 @@ import FrameBox from '@aziontech/webkit/frame-box'
import ItemList from '@aziontech/webkit/item-list'
import DocItem from '@aziontech/webkit/doc-item'

`Azion.AI.run` is the binding a [function](/en/documentation/platform/functions/) calls to invoke a model on [AI Inference](/en/documentation/platform/ai-inference/). It is a global of [Azion Runtime](/en/documentation/devtools/runtime/), so a function reaches it with no import line and no credential. The binding carries the model id and the request body. The fields that body accepts, and the object the model returns, are the same over the HTTP endpoint and are documented in [Model invocation](/en/documentation/platform/ai-inference/model-invocation/).
`Azion.AI.run` is the binding a [Function](/en/documentation/platform/functions/) calls to invoke a model on [AI Inference](/en/documentation/platform/ai-inference/). It is a global of [Azion Runtime](/en/documentation/devtools/runtime/), so a Function reaches it with no import line and no credential. The binding carries the model id and the request body. The fields that body accepts, and the object the model returns, are the same over the HTTP endpoint and are documented in [Model invocation](/en/documentation/platform/ai-inference/model-invocation/).

:::note
Under `azion dev`, `Azion.AI` is `undefined`, and a call to `Azion.AI.run` throws `TypeError: Cannot read properties of undefined (reading 'run')`. Test model calls on a deployed function.
Under `azion dev`, `Azion.AI` is `undefined`, and a call to `Azion.AI.run` throws `TypeError: Cannot read properties of undefined (reading 'run')`. Test model calls on a deployed Function.
:::

---

## Parameters

`Azion.AI.run` takes two parameters, and the call is asynchronous: a function awaits it and reads the model response from the resolved value.
`Azion.AI.run` takes two parameters, and the call is asynchronous: a Function awaits it and reads the model response from the resolved value.

| Parameter | Type | Description |
| --- | --- | --- |
| Model id | string | The id of the model to invoke, in the form its model page states. |
| Request body | object | The request fields for the operation. The model id is not repeated inside it. |
| Request body | object | The request fields for the operation. A `model` field inside it is ignored, and so is a field the model does not know. |

Model ids differ in form between models, so copy the id from the model's own page in [AI models](/en/documentation/platform/ai-inference/models/) rather than deriving it from the model name.

---

## Call a model

A call to a chat model inside a function:
A call to a chat model inside a Function:

```ts
const modelResponse = await Azion.AI.run("Qwen/Qwen3-30B-A3B-Instruct-2507-FP8", {
Expand All @@ -52,7 +52,19 @@ return modelResponse

## Return value

The resolved value is the object the model returns. A chat model returns a `chat.completion` object, an embedding model returns a `list` of embeddings, and a reranking model returns the ranked results. The generated text of a chat model sits at `modelResponse?.choices?.[0]?.message?.content`, which is the content of the first entry in `choices`. Every field of the three shapes is listed in [Model invocation](/en/documentation/platform/ai-inference/model-invocation/#response-fields).
The resolved value is the object the model returns. A chat model returns a `chat.completion` object, an embedding model returns a `list` of embeddings, and a reranking model returns the ranked results. With `stream` set to `true` in the body, the call resolves to an async iterator of `chat.completion.chunk` objects instead, which the Function reads with `for await`. The `model` field of a response is the account's internal id for the model, which can differ from the id the call passed. The generated text of a chat model sits at `modelResponse?.choices?.[0]?.message?.content`, which is the content of the first entry in `choices`. Every field of the three shapes is listed in [Model invocation](/en/documentation/platform/ai-inference/model-invocation/#response-fields).

---

## List models

`Azion.AI.models()` returns the ids of the models your account can call, as an array of strings. It takes no argument, and the call is asynchronous:

```ts
const modelIds = await Azion.AI.models()
```

The list can hold two forms of one model's id, a repository path and a lowercase form, and either one selects that model in `Azion.AI.run`.

---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,10 @@ The table lists each module with its status. Where a module page exists, the mod
- 🟡 **Partially supported**: only some methods or features are available.
- 🔴 **Build-only**: the module is included so the build completes without errors, but its API calls do not work at runtime.

:::note[About the Azion Cells runtime environment]
A function on Azion runs inside a **Cell**, the isolation environment Azion builds on **V8 isolates**. A Cell is a lightweight, secure execution context, and the code inside it has no direct access to the operating system underneath.
:::note[About the isolation environment]
A function on Azion runs inside an isolation environment that Azion builds on **V8 isolates**. It is a lightweight, secure execution context, and the code inside it has no direct access to the operating system underneath.

For multi-tenant security and infrastructure integrity, a Cell does not give a function:
For multi-tenant security and infrastructure integrity, the isolation environment does not give a function:

- The properties of the machine that runs it. `node:os` and `process` return fixed values instead, such as `linux` for `os.platform()`, an empty string for `os.hostname()`, and `0` for `os.totalmem()`.
- DNS resolution through native system calls (`node:dns`, `node:dns/promises`).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ To read the output from the terminal with Azion CLI:
azion logs cells --tail
```

The terminal prints the console messages of the last 5 minutes and keeps printing new ones. Add `--function-id` to restrict the output to one function. The subcommand is named `cells` because a function runs inside a Cell, the isolation environment Azion builds on V8 isolates.
The terminal prints the console messages of the last 5 minutes and keeps printing new ones. Add `--function-id` to restrict the output to one function.

To send the same output to an endpoint you own:

Expand Down
Loading
Loading