Streaming Responses

For real-time token generation, set stream: true in your request. The response will be delivered as Server-Sent Events (SSE).

Request:

{
  "model": "text-prime",
  "messages": [{"role": "user", "content": "Hello!"}],
  "stream": true
}

Response Format:

Each SSE event contains a data: field with a JSON chunk:

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1677858242,"model":"text-prime","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1677858242,"model":"text-prime","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1677858242,"model":"text-prime","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1677858242,"model":"text-prime","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Parsing tips:

• Each chunk is prefixed with data:

• The stream ends with data: [DONE]

• Collect delta.content from each chunk to build the full response

• Check finish_reason to detect completion (stop, length, tool_calls)

Python

import requests

response = requests.post(
    'https://api.cortex.thegrid.ai/api/v1/chat/completions',
    headers={
        'Authorization': 'Bearer YOUR_API_KEY',
        'Content-Type': 'application/json'
    },
    json={
        'model': 'text-prime',
        'messages': [{'role': 'user', 'content': 'Hello!'}],
        'stream': True
    },
    stream=True
)

for line in response.iter_lines():
    if line:
        line = line.decode('utf-8')
        if line.startswith('data: ') and line != 'data: [DONE]':
            import json
            chunk = json.loads(line[6:])
            content = chunk['choices'][0]['delta'].get('content', '')
            print(content, end='', flush=True)

JavaScript

OpenAI SDK