Retain and Assign Site to Inventory Numbers

We have many sites that have their own set of purchased and reserved numbers. If we move the numbers to inventory, I don't see a way to assign a "company" number to a site once in inventory. If I assign a spare number to the site it belongs to it shows up as the selected site but loses the site and goes to "Company" when brought back into inventory. We are moving numbers from Auto Receptionist to get rid of the long list of numbers available in CallerID list. Some of our inventory numbers have a site set and I can filter by site in inventory list but can't set a number to a specific site once in inventory.