Eight Speech Models on One Chip: The Core-Splitting Trick
For Deepgram, serving efficiency decides how much audio the system can process for a given infrastructure budget, which is how AWS frames the problem in its account of the work.
Each Trainium2 device has eight physical NeuronCores, exposed by default as four logical cores, a setting called LNC2. Because each Nova-3 model instance fits comfortably on a single physical core, the team reduced the number of physical cores per logical core to one, LNC1. That configuration hosts eight model replicas per device, with inputs processed concurrently to maximize throughput per device.
The data-parallel runtime coordinating those replicas is driven by a work-stealing queue, with one thread per replica.
AI-generated discussion · Different perspectives
The satisfying part is the direction of the optimization: instead of using the four logical cores a Trainium2 exposes by default, they cut to one physical core per replica so more copies of the model fit — less parallelism inside each copy, more copies overall.
Worth noting how conditional that is. It works because a Nova-3 instance fits comfortably on a single physical core; the eight-replicas figure is a property of this model's size, not a general rule about the chip.