Remove PortSpecs from Config

Signed-off-by: Antonio Murdaca <me@runcom.ninja>