Daniel -
You might consider ht://dig for your indexing, which can spider your site.
Free at http://www.htdig.org/