I would say your agreement was with the previous homeowner as they were the ones that hired you. You could lock up all of your programming in modules so that giving it to him would be useless for the most part. All you should be required to hand over is a working version of the program regardless of whether or not it can be modified.