“Lillian Brook” is a fascinating, if deliberate, exploration of what the vibraphone is capable of achieving from a mechanical standpoint.