Documentation

Overview

ISON is a minimal, human-readable data serialization format optimized for Large Language Models. It achieves 30-70% token reduction compared to JSON by using tabular structures and eliminating redundant punctuation.

Why ISON?

• Token Efficiency - Fit 2× more data in the same context window
• LLM-Friendly - Tabular format matches training data patterns
• Human-Readable - Clean, minimal syntax without visual noise
• Graph Support - Native references for relationships
• Type Safe - Automatic type inference with explicit typing when needed

When to Use ISON

• RAG pipelines with large document chunks
• Multi-agent system communication
• Database query results (SQL, vector, graph)
• LLM prompts with structured context
• Tool/function call results
• Knowledge base serialization

Installation

Python

pip install ison-py

Requirements

• Python 3.10 or higher
• No external dependencies

Verify Installation

python -c "import ison_parser; print(ison_parser.__version__)"

Quick Start

Basic Example

import ison_parser

# Parse ISON text
ison_text = """
table.users
id name email active
1 Alice alice@example.com true
2 Bob bob@example.com false
"""

doc = ison_parser.loads(ison_text)

# Access data
users = doc['users']
for row in users.rows:
    print(f"{row['name']}: {row['email']}")

# Convert to JSON
json_output = doc.to_json()
print(json_output)

Creating ISON from Python Data

import ison_parser

data = {
    'products': [
        {'id': 1, 'name': 'Widget', 'price': 29.99},
        {'id': 2, 'name': 'Gadget', 'price': 49.99}
    ]
}

# Convert to ISON
doc = ison_parser.from_dict(data)
ison_text = ison_parser.dumps(doc)
print(ison_text)

Output:

table.products
id name    price
1  Widget  29.99
2  Gadget  49.99

Syntax Reference

Block Structure

An ISON document consists of one or more blocks:

kind.name
field1 field2 field3
value1 value2 value3
value1 value2 value3

Block Kinds

Data Types

Type Annotations

Field headers can include optional type hints:

table.products
id:int name:string price:float in_stock:bool
1 Widget 29.99 true
2 Gadget 49.99 false

Supported types: int, float, string, bool, ref, computed, node, edge

Relationship-Typed References

References can include relationship types (UPPERCASE) for graph semantics:

table.users
id name team
101 Mahesh :MEMBER_OF:10
102 John :LEADS:10

This enables LLMs to understand the nature of relationships, perfect for knowledge graphs.

Summary Rows

Tables can include summary rows after a --- separator:

table.sales
region q1 q2 total
Americas 1.2M 1.4M 2.6M
EMEA 0.8M 0.9M 1.7M
---
TOTAL 2.0M 2.3M 4.3M

Quoting Rules

Strings must be quoted if they contain:

• Spaces or tabs
• Reserved words: true, false, null
• Look like numbers: "123", "3.14"
• Start with colon: ":tag"

Escape Sequences

References

Express relationships between records:

table.teams
id name
10 Engineering
20 Marketing

table.employees
id name team
1 Alice :10
2 Bob :20

Nested Fields

Use dot-path notation for hierarchical data:

object.order
id customer.name customer.address.city total
1001 Alice "New York" 125.50

Comments

# This is a comment
table.users
id name
1 Alice  # Comments can appear anywhere

Python API Reference

doc = ison_parser.loads(ison_text)

doc = ison_parser.load('data.ison')

ison_text = ison_parser.dumps(doc)

ison_parser.dump(doc, 'output.ison')

doc = ison_parser.from_dict({'users': [...]})  

Document Object

users_block = doc['users']

Block Object

Reference Object

ref = Reference(id="10")
ref = Reference(id="101", type="user")

TypeScript API Reference

Full TypeScript implementation with complete type definitions.

npm install ison-js
# or
yarn add ison-js

Basic Usage

import { parse, dumps, fromDict, Reference } from 'ison-js';

// Parse ISON text
const isonText = `
table.users
id:int name:string email active:bool
1 Alice alice@example.com true
2 Bob bob@example.com false
`;

const doc = parse(isonText);

// Access blocks
const users = doc.getBlock('users');
console.log(`Users: ${users.size()}`);

// Access values
for (const row of users.rows) {
    console.log(`${row.id}: ${row.name}`);
}

// Convert to JSON
const json = doc.toJson();

// Serialize back to ISON
const isonOutput = dumps(doc);

Parsing Functions

const doc = parse(text);
const doc = loads(text);  // Alias

const doc = loadsIsonl(isonlText);

const data = {
    products: [
        { id: 1, name: 'Widget', price: 29.99 },
        { id: 2, name: 'Gadget', price: 49.99 }
    ]
};
const doc = fromDict(data);

Serialization

import { dumps, dumpsIsonl } from 'ison-js';

const ison = dumps(doc);           // With column alignment
const ison = dumps(doc, false);    // Without alignment
const isonl = dumpsIsonl(doc);     // ISONL format
const json = doc.toJson();         // JSON string

Type Guards

import { isNull, isBool, isInt, isFloat, isString, isReference } from 'ison-js';

if (isReference(value)) {
    const ref: Reference = value;
    ref.id;              // Referenced ID
    ref.type;            // Optional type/namespace
    ref.isRelationship(); // true if UPPERCASE type
}

Rust API Reference

Safe Rust implementation with optional serde integration.

# Add to Cargo.toml
[dependencies]
ison-rs = "1.0"

# With serde/JSON support (default)
ison-rs = { version = "1.0", features = ["serde"] }

Basic Usage

use ison_parser::{parse, dumps, Value};

fn main() -> Result<(), ison_parser::ISONError> {
    let ison_text = r#"
table.users
id:int name:string email active:bool
1 Alice alice@example.com true
2 Bob bob@example.com false
"#;

    let doc = parse(ison_text)?;
    let users = doc.get("users").unwrap();

    for row in &users.rows {
        let name = row.get("name").and_then(|v| v.as_str()).unwrap_or("");
        println!("{}", name);
    }

    // Serialize back
    let output = dumps(&doc, true);
    Ok(())
}

Parsing Functions

let doc = parse(text)?;
let doc = loads(text)?;  // Alias

let doc = loads_isonl(isonl_text)?;

Value Types

use ison_parser::Value;

// Value is an enum
match value {
    Value::Null => {},
    Value::Bool(b) => {},
    Value::Int(i) => {},
    Value::Float(f) => {},
    Value::String(s) => {},
    Value::Reference(r) => {},
}

// Type checking
value.is_null();
value.is_int();
value.is_reference();

// Value extraction (returns Option)
let i: Option<i64> = value.as_int();
let s: Option<&str> = value.as_str();
let r: Option<&Reference> = value.as_reference();

References

use ison_parser::Reference;

let simple = Reference::new("42");
let typed = Reference::with_type("101", "user");
let rel = Reference::with_type("10", "MEMBER_OF");

rel.is_relationship();      // true
rel.relationship_type();    // Some("MEMBER_OF")

JSON Export (requires serde feature)

let json = doc.to_json(true);   // Pretty-printed
let json = doc.to_json(false);  // Compact

C++ API Reference

A header-only C++11 implementation compatible with llama.cpp and other C++ projects.

# Option 1: Copy header
cp ison_parser.hpp /your/project/include/

# Option 2: CMake
add_subdirectory(ison-cpp)
target_link_libraries(your_target PRIVATE ison::parser)

Basic Usage

#include "ison_parser.hpp"
using namespace ison;

// Parse ISON string
std::string ison_text = R"(
table.users
id:int name:string email active:bool
1 Alice alice@example.com true
2 Bob bob@example.com false
)";

auto doc = parse(ison_text);

// Access blocks
auto& users = doc["users"];
std::cout << "Users: " << users.size() << std::endl;

// Access values with type checking
for (const auto& row : users.rows) {
    int64_t id = as_int(row.at("id"));
    const std::string& name = as_string(row.at("name"));
    bool active = as_bool(row.at("active"));
}

// Convert to JSON
std::string json = doc.to_json();

// Serialize back to ISON
std::string ison_output = dumps(doc);

Parsing Functions

Document doc = ison::parse(text);
Document doc = ison::loads(text);  // Alias

Document doc = ison::load("data.ison");

Document doc = ison::loads_isonl(isonl_text);

Serialization Functions

std::string ison = ison::dumps(doc);
std::string ison = ison::dumps(doc, false);  // No alignment

std::string isonl = ison::dumps_isonl(doc);

std::string json = doc.to_json();
std::string json = doc.to_json(4);  // Custom indent

Type Checking & Value Extraction

// Type checking
is_null(value);
is_bool(value);
is_int(value);
is_float(value);
is_string(value);
is_reference(value);

// Value extraction (throws on wrong type)
bool b = as_bool(value);
int64_t i = as_int(value);
double d = as_float(value);
const std::string& s = as_string(value);
const Reference& r = as_reference(value);

References

// Simple reference :42
Reference ref("42");
ref.id;           // "42"
ref.to_ison();    // ":42"

// Namespaced reference :user:101
Reference ref("101", "user");
ref.get_namespace(); // "user"
ref.is_relationship(); // false

// Relationship reference :MEMBER_OF:10
Reference ref("10", "MEMBER_OF");
ref.is_relationship();     // true
ref.relationship_type();   // "MEMBER_OF"

Building

# CMake
mkdir build && cd build
cmake ..
make
./test_ison_parser

# Manual (GCC/Clang)
g++ -std=c++11 -I include your_file.cpp -o your_app

# Visual Studio
cmake .. -G "Visual Studio 17 2022"
cmake --build . --config Release

Requirements: C++11 minimum (GCC 4.8+, Clang 3.3+, MSVC 2015+). Auto-upgrades to use std::optional when C++17 is available.

Command Line Interface

Validate ISON File

python ison_parser.py data.ison --validate

Convert ISON to JSON

python ison_parser.py data.ison -f json -o output.json

Normalize/Format ISON File

python ison_parser.py data.ison -f ison -o clean.ison

Parse from stdin

cat data.ison | python ison_parser.py -f json

CLI Options

ISONL Streaming Format

ISONL (ISON Lines) is a line-based streaming format for ISON. Each line is a self-contained record, enabling incremental parsing, streaming workflows, and append-only storage.

Format

Each ISONL line has three pipe-separated sections:

kind.name|field1 field2 field3|value1 value2 value3

Example

# Fine-tuning dataset
table.sft|instruction response|"Summarize this" "Brief summary..."
table.sft|instruction response|"Translate to Spanish" "Hola, mundo"

# Event stream with references
table.events|ts type user|10:30:00 click :1
table.events|ts type user|10:30:05 pageview :1

Python API

from ison_parser import loads_isonl, dumps_isonl, isonl_stream

# Parse ISONL text
doc = loads_isonl(isonl_text)

# Convert to ISONL
isonl_text = dumps_isonl(doc)

# Stream large files (constant memory)
with open("large_dataset.isonl", "r") as f:
    for record in isonl_stream(f):
        process(record)

Conversion

from ison_parser import ison_to_isonl, isonl_to_ison

# ISON -> ISONL (one line per record)
isonl = ison_to_isonl(ison_text)

# ISONL -> ISON (grouped blocks)
ison = isonl_to_ison(isonl_text)

See the full ISONL documentation for more details.

LLM Integration

OpenAI Integration

import openai
import ison_parser

def query_with_context(question, context_data):
    # Convert context to ISON
    ison_context = ison_parser.dumps(
        ison_parser.from_dict(context_data)
    )

    response = openai.chat.completions.create(
        model="gpt-4",
        messages=[
            {
                "role": "user",
                "content": f"""Context:
{ison_context}

Question: {question}

Answer based on the context provided."""
            }
        ]
    )

    return response.choices[0].message.content

Anthropic Claude Integration

import anthropic
import ison_parser

client = anthropic.Anthropic()

def query_with_context(question, context_data):
    ison_context = ison_parser.dumps(
        ison_parser.from_dict(context_data)
    )

    message = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": f"""Context:
{ison_context}

Question: {question}"""
            }
        ]
    )

    return message.content[0].text

RAG Pipeline Example

import ison_parser
from vector_db import search_similar

def rag_query(question: str, top_k: int = 5):
    # 1. Retrieve similar documents
    results = search_similar(question, top_k=top_k)

    # 2. Format as ISON (30-70% fewer tokens than JSON!)
    context = {
        "documents": [
            {
                "id": r.id,
                "content": r.content,
                "score": r.score,
                "metadata": r.metadata
            }
            for r in results
        ]
    }

    ison_context = ison_parser.dumps(
        ison_parser.from_dict(context)
    )

    # 3. Query LLM with compact context
    return query_llm(question, ison_context)

Framework Integrations

ISON provides official integrations with popular LLM frameworks and tools. All integrations support ISONantic for type-safe validation.

# All integrations included with ison-py
pip install ison-py

# Optional dependencies for specific frameworks
pip install ison-py[langchain]    # LangChain
pip install ison-py[llamaindex]   # LlamaIndex
pip install ison-py[mcp]          # MCP Server/Client
pip install ison-py[all]          # All integrations

Available Integrations

LangChain Integration

from ison_parser.integrations import ISONOutputParser, ISONanticOutputParser
from langchain_openai import ChatOpenAI

# Basic ISON parsing
parser = ISONOutputParser()
llm = ChatOpenAI(model="gpt-4")
chain = llm | parser

result = chain.invoke("List 3 users in ISON format")

# With ISONantic validation
from isonantic import TableModel, Field

class User(TableModel):
    id: int = Field(description="User ID")
    name: str = Field(description="User name")
    email: str = Field(description="Email address")

parser = ISONanticOutputParser(model=User)
chain = llm | parser
users = chain.invoke("Generate 3 users")

LlamaIndex Integration

from ison_parser.integrations import ISONReader, ISONNodeParser
from llama_index.core import VectorStoreIndex

# Load ISON files for RAG
reader = ISONReader()
documents = reader.load_data("data.ison")

# Create index
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What users are active?")

MCP Server/Client

# Start MCP server
from ison_parser.integrations import ISONMCPServer

server = ISONMCPServer()
server.run()  # Exposes parse_ison, validate_ison, query_ison tools

# Use MCP client
from ison_parser.integrations import ISONMCPClient

async with ISONMCPClient() as client:
    result = await client.parse_ison(ison_text)
    validated = await client.validate_ison(ison_text, schema)

OpenAI Function Calling

from ison_parser.integrations import OpenAIISONTools
import openai

tools = OpenAIISONTools()

response = openai.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Parse this ISON data"}],
    tools=tools.get_tools(),
    tool_choice="auto"
)

# Parse response with ISONantic model
result = tools.parse_response_typed(response)

Anthropic Tool Use

from ison_parser.integrations import AnthropicISONTools
import anthropic

tools = AnthropicISONTools()
client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    tools=tools.get_tools(),
    messages=[{"role": "user", "content": "Format this as ISON"}]
)

result = tools.parse_response_typed(response)

See the full integrations documentation for detailed examples and API reference.

SQL Database Plugins

ISON provides built-in plugins to export data from SQL databases directly to ISON format, making it easy to build Database-to-LLM pipelines.

# SQLite (zero dependencies - included)
pip install ison-py

# PostgreSQL
pip install ison-py[postgresql]

# SQLAlchemy (MySQL, MariaDB, Oracle, MS SQL)
pip install ison-py[sqlalchemy]

# All database plugins
pip install ison-py[databases]

SQLite Plugin

Zero-dependency SQLite support with automatic foreign key detection.

from ison_parser.plugins import SQLiteToISON

# Connect to database
exporter = SQLiteToISON('app.db')

# Export entire database
ison_text = exporter.export_all()

# Export specific tables
ison_text = exporter.export_tables(['users', 'orders'])

# Export custom query results
ison_text = exporter.export_query(
    "SELECT * FROM users WHERE active = 1",
    name="active_users"
)

# Stream large tables (constant memory)
for line in exporter.stream_table('logs', batch_size=1000):
    process(line)  # ISONL format

Output Example

table.users
id name email active
1 Alice alice@example.com true
2 Bob bob@example.com true

table.orders
id user_id product total
1001 :1 Widget 29.99
1002 :2 Gadget 49.99

Note: Foreign keys are automatically converted to ISON references (e.g., :1)

PostgreSQL Plugin

Full PostgreSQL support with server-side cursors for large datasets.

from ison_parser.plugins import PostgreSQLToISON

# Connect
exporter = PostgreSQLToISON(
    host='localhost',
    database='myapp',
    user='postgres',
    password='secret'
)

# Or use connection string
exporter = PostgreSQLToISON(
    dsn='postgresql://user:pass@host:5432/db'
)

# Export with relationships preserved
ison_text = exporter.export_all()

# Custom query
ison_text = exporter.export_query("""
    SELECT u.name, COUNT(o.id) as order_count
    FROM users u
    LEFT JOIN orders o ON u.id = o.user_id
    GROUP BY u.id
""", name="user_stats")

# Stream large tables
for line in exporter.stream_table('events'):
    process(line)

SQLAlchemy Plugin

Generic SQL support for any SQLAlchemy-compatible database.

from ison_parser.plugins import SQLAlchemyToISON

# MySQL
exporter = SQLAlchemyToISON('mysql+pymysql://user:pass@host/db')

# MariaDB
exporter = SQLAlchemyToISON('mariadb+pymysql://user:pass@host/db')

# MS SQL Server
exporter = SQLAlchemyToISON('mssql+pyodbc://user:pass@host/db')

# Export tables
ison_text = exporter.export_all()

# Export ORM models
from myapp.models import User, Order

ison_text = exporter.export_models([User, Order])

# Custom query with parameters
ison_text = exporter.export_query(
    "SELECT * FROM products WHERE price > :min_price",
    params={'min_price': 50},
    name="premium_products"
)

Quick Functions

from ison_parser.plugins.sqlite_plugin import sqlite_to_ison
from ison_parser.plugins.postgresql_plugin import postgresql_to_ison

# One-liner exports
ison = sqlite_to_ison('app.db', tables=['users'])
ison = postgresql_to_ison('mydb', query="SELECT * FROM active_users")

Vector Database Plugins

Export vector search results to ISON format for RAG (Retrieval-Augmented Generation) pipelines. These plugins are optimized to produce compact, token-efficient context for LLMs.

# ChromaDB
pip install ison-py[chroma]

# Pinecone
pip install ison-py[pinecone]

# Qdrant
pip install ison-py[qdrant]

# All vector databases
pip install ison-py[vectordbs]

ChromaDB Plugin

Export ChromaDB collections for RAG context.

from ison_parser.plugins import ChromaToISON

# Connect to ChromaDB
exporter = ChromaToISON(path='./chroma_db')
# Or: exporter = ChromaToISON(host='localhost', port=8000)

# Export RAG context (recommended for LLMs)
ison_context = exporter.export_for_rag(
    collection='documents',
    query='What is ISON?',
    n_results=5
)

# Export search results
ison_text = exporter.export_query_results(
    collection='documents',
    query_texts=['machine learning', 'neural networks'],
    n_results=10
)

# Export entire collection
ison_text = exporter.export_collection('documents')

# Stream large collections as ISONL
for line in exporter.stream_collection('documents'):
    process(line)

RAG Context Output

table.context
rank:int score:float source content
1 0.892 intro.md "ISON is a token-efficient data format..."
2 0.845 syntax.md "The basic syntax uses tables..."
3 0.821 examples.md "Here's how to convert JSON..."

Pinecone Plugin

Export from Pinecone's managed vector service.

from ison_parser.plugins import PineconeToISON

# Connect (uses PINECONE_API_KEY env var if not provided)
exporter = PineconeToISON(api_key='your-api-key')

# Export query results
ison_text = exporter.export_query_results(
    index='my-index',
    query_vector=embedding,
    top_k=10,
    namespace='production'
)

# Export with text query (requires embedding function)
def get_embedding(text):
    # Your embedding model here
    return openai.embeddings.create(input=text, model='text-embedding-3-small').data[0].embedding

ison_context = exporter.export_for_rag(
    index='knowledge-base',
    query='How do I configure authentication?',
    embedding_fn=get_embedding,
    top_k=5
)

# Fetch specific vectors by ID
ison_text = exporter.export_vectors(
    index='my-index',
    ids=['doc1', 'doc2', 'doc3']
)

# Get index statistics
ison_stats = exporter.export_namespace_stats('my-index')

Qdrant Plugin

Export from Qdrant vector database.

from ison_parser.plugins import QdrantToISON

# Connect to Qdrant
exporter = QdrantToISON(host='localhost', port=6333)
# Or cloud: exporter = QdrantToISON(url='https://xxx.cloud', api_key='...')

# Export search results
ison_text = exporter.export_search_results(
    collection='documents',
    query_vector=embedding,
    limit=10,
    score_threshold=0.7
)

# Export with filters
ison_text = exporter.export_search_results(
    collection='documents',
    query_vector=embedding,
    filter={'must': [{'key': 'category', 'match': {'value': 'technical'}}]}
)

# Export RAG context
ison_context = exporter.export_for_rag(
    collection='knowledge',
    query='deployment best practices',
    embedding_fn=get_embedding,
    limit=5
)

# Export collection metadata
ison_meta = exporter.export_collection_info('documents')

# Stream collection as ISONL
for line in exporter.stream_collection('documents', batch_size=100):
    process(line)

Quick Functions

from ison_parser.plugins.chroma_plugin import chroma_rag_context
from ison_parser.plugins.pinecone_plugin import pinecone_rag_context
from ison_parser.plugins.qdrant_plugin import qdrant_rag_context

# One-liner RAG context generation
context = chroma_rag_context('docs', 'What is ISON?', n_results=5)
context = pinecone_rag_context('index', 'question', embedding_fn, top_k=5)
context = qdrant_rag_context('collection', 'query', embedding_fn, limit=5)

Complete RAG Pipeline Example

import openai
from ison_parser.plugins import ChromaToISON

# Initialize
chroma = ChromaToISON(path='./vector_db')

def answer_question(question: str) -> str:
    # 1. Get relevant context in ISON format
    context = chroma.export_for_rag(
        collection='knowledge_base',
        query=question,
        n_results=5
    )

    # 2. Send to LLM (30-70% fewer tokens than JSON!)
    response = openai.chat.completions.create(
        model='gpt-4',
        messages=[{
            'role': 'user',
            'content': f"""Context:
{context}

Question: {question}

Answer based on the context provided."""
        }]
    )

    return response.choices[0].message.content

# Use it
answer = answer_question("How do I optimize ISON for large datasets?")

RudraDB Plugin

Export data from RudraDB, a high-performance Rust-based embedded database with relationship/graph capabilities.

pip install ison-py rudradb
# or
pip install ison-py rudradb-trishul

Basic Usage

from ison_parser.plugins import RudraDBToISON

# Connect to database
exporter = RudraDBToISON(db_path='./my_database')

# Export entire database
ison_text = exporter.export_all()

# Export specific collections
ison_text = exporter.export_collections(['users', 'orders'])

# Export single collection with limit
ison_text = exporter.export_collection('users', limit=100)

Output Example

table.users
id name email active
1 Alice alice@example.com true
2 Bob bob@example.com true

table.orders
id user_id product price
101 :user:1 Widget 29.99
102 :user:2 Gadget 49.99

Note: Relationships are automatically converted to ISON references

Exporting Query Results

# Export query results
query_result = db.query("SELECT * FROM users WHERE active = true")
ison_text = exporter.export_query(query_result, name="active_users")

# Export with relationships included
ison_text = exporter.export_with_relationships(
    'users',
    relationship_types=['PURCHASED', 'FOLLOWS'],
    depth=2
)

Streaming Large Collections (ISONL)

# Stream large collections with constant memory
for line in exporter.stream_collection('events', batch_size=1000):
    process(line)  # ISONL format

# Output:
# table.events|id timestamp type|1 2024-01-01T10:00:00 click
# table.events|id timestamp type|2 2024-01-01T10:05:00 view

RAG Pipeline Integration

# Get optimized context for LLM
context = exporter.export_for_rag(
    collection='documents',
    query_vector=embedding,  # Optional vector search
    limit=5
)

# Returns ranked results optimized for LLM context
# table.context
# rank:int score:float content
# 1 0.95 "Document content here..."
# 2 0.89 "Another relevant document..."

Quick Functions

from ison_parser.plugins.rudradb_plugin import (
    rudradb_to_ison,
    rudradb_query_to_ison,
    rudradb_rag_context
)

# One-liner exports
ison = rudradb_to_ison('./mydb', collections=['users'])

# Query result to ISON
ison = rudradb_query_to_ison(db, query_result, name='results')

# RAG context
context = rudradb_rag_context('./mydb', 'documents', query_vector, limit=5)

Best Practices

1. Use ISON at the LLM Boundary

Convert to ISON right before sending to the LLM, not earlier in your pipeline. Keep JSON for internal service-to-service communication.

# Good
data = fetch_from_api()  # Returns JSON
ison = to_ison(data)     # Convert at LLM boundary
response = llm.query(ison)

# Not recommended
data = fetch_from_api_as_ison()  # Too early

2. Normalize Nested Arrays

Instead of nested arrays, use separate tables with references:

# Good - Separate tables
table.orders
id customerId total
1 :101 99.99

table.items
orderId name qty
1 Widget 2
1 Gadget 1

# Avoid - Deep nesting

3. Quote When Needed

Always quote strings that contain spaces or could be misinterpreted:

# Correct
table.users
name city
Alice "New York"
Bob Dallas

# Wrong - city will be two separate tokens
table.users
name city
Alice New York

4. Use Meaningful Block Names

# Good
table.active_users
table.recent_orders

# Less clear
table.data1
table.temp

5. Leverage References

Use references to avoid data duplication:

table.products
id name price
1 Widget 29.99
2 Gadget 49.99

table.orders
orderId productId qty
100 :1 2
101 :2 1

6. Add Comments for Context

# User activity from the last 30 days
table.user_activity
userId action timestamp
101 login 2025-01-15
102 purchase 2025-01-16

Migration Guide: JSON to ISON

Arrays of Objects

Before (JSON):

{
  "users": [
    {"id": 1, "name": "Alice", "active": true},
    {"id": 2, "name": "Bob", "active": false}
  ]
}

After (ISON):

table.users
id name active
1 Alice true
2 Bob false

Nested Objects

Before (JSON):

{
  "order": {
    "id": 1001,
    "customer": {
      "name": "Alice",
      "address": {
        "city": "New York",
        "state": "NY"
      }
    }
  }
}

After (ISON):

object.order
id customer.name customer.address.city customer.address.state
1001 Alice "New York" NY

Related Data

Before (JSON):

{
  "orders": [
    {
      "id": 1,
      "customer": {"id": 101, "name": "Alice"},
      "total": 99.99
    }
  ]
}

After (ISON):

table.customers
id name
101 Alice

table.orders
id customerId total
1 :101 99.99

Automated Migration

import json
import ison_parser

# Load JSON
with open('data.json') as f:
    data = json.load(f)

# Convert to ISON
doc = ison_parser.from_dict(data)
ison_text = ison_parser.dumps(doc)

# Save
with open('data.ison', 'w') as f:
    f.write(ison_text)