If you are using a matrix with a 2-way driver, then yeah you could do it.  If not, then you would have to build flags for every source and every input, then have a button become 'visible' on an XP-8 flag if you select a source that's already in use (or however you want it to work).  With a pre-built driver it gets exponentially easier.

Yes you can do it without a driver, but it's going to be fun........