You’ve experienced the power of large language models (LLMs) if you’ve used Copilot to answer complex questions. The models are so large that they can require significant computing resources to run, making the rise of small language models (SLMs) a big deal.
SLMs are still quite large with several billion parameters — in contrast to hundreds of billions of parameters in LLMs — but they’re small enough to run on a phone offline. Parameters are variables, or adjustable elements, that determine a model’s behavior.
“Small language models can make AI more accessible due to their size and affordability,” says Sebastien Bubeck, who leads the Machine Learning Foundations group at Microsoft Research. “At the same time, we’re discovering new ways to make them as powerful as large language models.”
Microsoft researchers have developed and released two SLMs — Phi and Orca— that perform as well as or better than large language models in certain areas, challenging the notion that scale is required for performance.
Unlike LLMs trained on vast amounts of internet data, the smaller models use curated, high-quality training data, with researchers finding new thresholds for size and performance. This year, you can expect to see improved models designed to foster more research and innovation.