Go Deep, or Go Wide?