It's not as obvious as it ought to be (in fact, it's anti-obvious): Inst Group (button at bottom) -> (select the layout to be deleted) -> Customize -> Select Layout -> Delete Active Layout
Just had to start with the assumption that if there's a way to increase the number of layouts then there's got to be away to decrease them. Only took about 10 seconds of poking to find it.