Under the Hood

What it is: The internal storage system that handles every data request behind ChatGPT and other OpenAI products: logins, conversation history, settings, and more.

What's new: OpenAI has published a detailed account of how Habitat scaled from a simple code library at launch in late 2023 to a system handling over 70 million requests per second across nearly 40 regions globally, according to OpenAI's engineering blog.

How it works: Habitat sits between product code and the underlying database (Azure Cosmos DB) and handles the routing, permissions, encryption, and caching so that product engineers never need to manage those directly. When a user opens a chat, Habitat fields multiple simultaneous data lookups, resolves where each piece of information lives, and returns it fast enough that the product feels instant. It now stores more than 500 petabytes (a petabyte is roughly a million gigabytes) of data.

What it can't do: This is part one of a two-part post. The second part, covering multi-tenant reliability, read optimisation, and the full Azure Cosmos DB integration, has not yet been published.

Who should care: Anyone curious about what it actually takes to keep a billion-user product running reliably. The engineering tradeoffs here are a useful counterpoint to AI coverage that focuses only on model capabilities.

Try it / read it: OpenAI engineering blog

What it is: A fully assembled image-generation workspace built on Gradio's new Workflow system, replicating most of the features of AUTOMATIC1111's stable-diffusion-webui — a popular open-source tool for generating images with Stable Diffusion models.

What's new: Previous Gradio tools required writing code to connect models together. Workflow1111 replaces that with a visual canvas where models, functions, and data sources are laid out as nodes you connect by drawing edges between them.

How it works: The canvas contains eleven media pipelines built from seventy-three individual nodes. Each node wraps one operation — a Python function (fn), a model call, another Gradio Space, or a dataset row. Connecting a node's output port to another node's input port defines how data flows. Because nodes with shared inputs run simultaneously, tasks like image captioning and label classification happen in parallel rather than waiting in sequence.

What it can't do: Running the pipelines uses your own Hugging Face account quota, so heavy use will hit rate limits. The post does not address performance on large or high-resolution generation tasks, and the canvas approach may be harder to debug than code when something breaks unexpectedly.

Who should care: Anyone building or experimenting with image generation workflows who wants to skip the setup overhead of wiring models together manually.

Try it / read it: Workflow1111 on Hugging Face

What it is: A research paper studying how to make AI safety filters refuse only the genuinely harmful slice of a topic, rather than blocking the whole subject.

What's new: Most safety systems work at the topic level — flag "politics" or "weapons" and block everything in that category. This paper argues that is too blunt and too leaky at the same time. A civics tutor needs to answer factual election questions but refuse requests to write targeted political manipulation. A topic-level filter cannot express that distinction.

How it works: The researchers define a narrow target: within political prompts, only refuse requests for manipulation or persuasion, while continuing to answer factual questions. They train the model using self-distillation (where the model generates its own training examples of correct refusals) but fix a specific flaw in that pipeline. Standard single-attempt generation silently drops prompts where no good refusal example is produced — in their tests, that was nearly 20% of harmful prompts. Their fix retries those failed cases with progressively stronger steering, bringing failures down from 19.88% to 0.20%. The result is a model that can hold a sharper line between "answer this" and "refuse this" within the same topic.

What it can't do: The paper tests only on political persuasion as its harmful category. How well this approach transfers to other topics — medical advice, weapons, financial fraud — is untested. The authors acknowledge the boundary between harmful and benign prompts will vary by deployment context, and that defining it correctly is its own unsolved problem.

Who should care: Teams building AI products for regulated or sensitive industries where blanket topic refusals create as many problems as they solve.

Read it: Hugging Face blog

What it is: A method of hiding text inside an email so humans cannot see it, but computers can read it perfectly.

What's new: ASCII smuggling started as a way to attack AI agents. Spammers have now adopted it to fool spam filters instead. Microsoft reports that detections jumped from 21,000 per day to 2.5 million within four days of a campaign starting in February, according to Ars Technica.

How it works: Standard text is encoded in ASCII (American Standard Code for Information Interchange — the foundational system that maps letters to numbers computers understand). Unicode, the broader modern standard, includes a tag range that mirrors ASCII almost exactly, but those tag characters are invisible when rendered on screen. A spammer writes banned keywords like "free money" or "click here" using these invisible characters. A human moderator reviewing the email sees nothing suspicious. The spam filter, processing raw text, may not catch the trick either. Think of it as writing a message in ink that only certain scanners can detect — and the security guard is using the wrong scanner.

What it can't do: Filters are already being updated. Microsoft's own Defender tool is what caught and measured this campaign, so the technique's effectiveness will shrink as detection improves.

Who should care: Anyone who manages email for a small business, or wonders why spam occasionally slips past filters that should have caught it.

Read it: Ars Technica

What it is: A new way for Google's Gemini models to analyse video by deciding, on the fly, which parts of a video to actually watch — rather than processing every frame at a fixed rate.

What's new: Standard video processing ingests footage at a fixed pace, typically one frame per second. That forces developers to trade off between cost and detail. Agentic video understanding lets the model seek through footage actively, inspecting only the segments relevant to the task.

How it works: Instead of receiving a uniform stream of frames, Gemini enters an agentic loop — a process where it repeatedly calls an internal tool to fetch specific video segments, audio, or transcript excerpts based on what it needs to answer the query. Think of it like a researcher who skims a report and only reads the footnotes when they become relevant, rather than transcribing the whole document word for word. According to Google's own benchmarks, this cuts analysis costs by up to 66% and token consumption (the volume of data the model processes, which drives cost) by up to 88%, while improving accuracy by up to 7%.

What it can't do: The benchmark figures come from Google's own testing, not independent evaluation. The gains are most pronounced on long videos; performance on short clips is less dramatically different from static processing. Frontier testing beyond the three supported Flash models has not been published.

Who should care: Developers building tools that process long-form video — surveillance, lecture transcription, automated video editing — stand to see meaningful cost reductions today.

Try it / read it: Google DeepMind blog

What it is: A new evaluation dataset for automatic speech recognition (ASR — software that converts spoken words into text) covering Indian English and Hindi, added to the Open ASR Leaderboard (a public ranking of how accurately different speech-to-text models perform).

What's new: Hindi is the first Indic language on a leaderboard that previously covered only European languages — despite Hindi being spoken by more than half a billion people. Most ASR benchmarks measure average accuracy across a generic population; Monsoon is built to show where accuracy breaks down, and for whom.

How it works: Researchers collected unscripted conversations across hundreds of districts in India rather than centralised recording studios. Speakers used their own phones in real environments — indoors, outdoors, varying mobile connections. Each audio clip is tagged with 12 attributes: age, gender, geography, device, education level, income band, and more. This means a model's word error rate (WER — the percentage of words it gets wrong) can be broken down by speaker group rather than reported as a single average that hides who gets worse results.

What it can't do: The Hindi dataset has no normaliser yet — software that treats spelling variants of the same word as correct. That makes comparison across models less reliable for Hindi than for English. The authors flag this as a known gap.

Who should care: Anyone building or choosing voice tools for multilingual or non-Western-English audiences, and anyone who wants AI benchmarks to reflect actual user diversity.

Try it / read it: Hugging Face blog

What it is: A technique for building AI search systems that match queries to documents token by token, rather than compressing everything into a single summary score.

What's new: The Sentence Transformers library now includes a MultiVectorEncoder class with a complete training pipeline, making it practical to fine-tune or build these models from scratch on a consumer GPU.

How it works: A standard search model compresses an entire document into one vector (think: a single number that summarises the whole page), then compares that summary to your query. A multi-vector model — also called a ColBERT-style or late-interaction model — keeps one small vector per word token instead. When you search, every word in your query finds its closest match in the document, and those individual scores are added up using an operation called MaxSim. This preserves fine-grained word-level signals that a single summary would lose. The trade-off is a larger index: storing one vector per token takes significantly more space than storing one per document.

The author trained a medical retrieval model in 14.5 hours on a single RTX 3090 GPU and reports it outperformed every general-purpose retrieval model tested, including dense, sparse, and lexical approaches, on a medical dataset where passages averaged 941 tokens.

What it can't do: The approach requires in-domain training data. Without it, you are back to a general-purpose model. Index size grows proportionally with document length, which becomes a practical storage and speed constraint at scale. Frontier-scale testing has not been published.

Who should care: Developers building search over legal, medical, financial, or internal documents where standard search returns inconsistent results.

Try it / read it: Hugging Face blog post

What it is: A training technique that produces a compressed, 4-bit AI model that outperforms the full-precision version it was derived from.

What's new: Most compressed models accept some accuracy loss as the cost of being smaller and cheaper to run. QAH inverts that: applied to a GPT-class 120-billion-parameter (think of parameters as the adjustable settings inside a model — more means more capacity) model compressed to 60 billion parameters and then quantized (converted from 16-bit to 4-bit numerical precision), the result beats the original 16-bit checkpoint on 7 of 9 benchmarks.

How it works: After structural compression and quantization, models are typically "healed" by fine-tuning against a task loss or by distillation — where a smaller student model is trained to mimic a larger teacher. The problem: once a model has been structurally compressed, there is no full-size version of the smaller architecture to learn from. QAH sidesteps this by distilling directly from the original, pre-compression model. The teacher and student do not share an architecture; what transfers is the teacher's output distribution, matched using a KL-divergence loss (a measure of how different two probability distributions are). This means the 4-bit student receives supervision the earlier recovery stage never provided.

What it can't do: Results are reported on one model family and one benchmark set. Frontier-scale generalisation is untested. The authors also flag that standard quantization-aware training can become unstable if run too long.

Who should care: Anyone deploying large models on constrained hardware — this suggests compression does not have to mean a capability penalty.

Read it: Hugging Face Blog

What it is: A set of small add-on models for Liquid AI's LFM2.5 language models that make them generate text significantly faster, without changing what they say.

What's new: Most AI inference (the process of generating a response from a trained model) is bottlenecked by memory, not processing power — the model has to load its weights (the numerical values learned during training) from slow memory on every single token it generates. DSpark cuts that cost using speculative decoding: a lightweight draft model guesses several tokens ahead, and the main model verifies all of them in one pass rather than generating one at a time.

How it works: DSpark combines three components. A parallel draft backbone (a small network conditioned on the main model's context) produces multiple candidate tokens at once. A sequential head then adds dependency between neighbouring tokens to improve how many candidates the main model accepts. Finally, a confidence-scheduled verifier predicts which candidate tokens are likely to survive and prunes the weakest ones before the main model spends time checking them.

What it can't do: Gains vary significantly by task. The 1.2B model sees speedup swing by as much as 52% depending on the text being generated. The 8B model shows only an 18% improvement on a Mac due to current limitations in the Metal backend used by llama.cpp. On-device speeds reach approximately 140 tokens per second on an M4 Max MacBook Pro, depending on the dataset.

Who should care: Developers running LFM2.5 models locally or on edge hardware where every millisecond of response time matters.

Try it / read it: Liquid AI on Hugging Face

What it is: A set of small, compressed AI models from Liquid AI, designed to run on everyday devices like laptops, phones, and Raspberry Pi boards.

What's new: Most compressed models lose meaningful accuracy in the process. Liquid AI's approach recovers nearly all of it while keeping the same small file size and fast speed as a standard compressed model.

How it works: The technique is called Quantization-Aware Distillation (QAD). Quantization (shrinking a model by reducing the numerical precision of its internal values, the way you'd compress a photo to a smaller file) typically costs accuracy. QAD counters this by running a full-precision "teacher" model alongside a compressed "student" model during training. The student learns to match the teacher's outputs precisely, rather than just being squashed down after the fact. The result is a model that fits in the same memory as a standard compressed file but behaves closer to the full-size original.

What it can't do: According to Liquid AI's own benchmarks — covering reasoning, instruction-following, and tool use across sets including GPQA Diamond (a graduate-level science question set) and MMLU-Pro — the QAD models recover around 97% of the accuracy lost through standard compression. Frontier-scale models were not tested; all four models here are small, ranging from 230 million to 2.6 billion parameters (the numerical values that define a model's behaviour).

Who should care: Developers and hobbyists who want a capable AI model running locally, without cloud costs or privacy trade-offs.

Try it / read it: Hugging Face — LFM2.5 QAD