Block schedule availablity in the link not with events

Currently I am able to add time slots at a weekly level for my scheduling link. However, I'd like to block out some specific dates which I know would be difficult due to some person engagements or because I'll be on the road between meetings. I can see an option to check my availability, but maintaining all that finer detail on a calendar is tough. So would like simple overrides for selected dates to remove my availability for those given dates.