I don't think what you describe is a bad idea but I would never go to that level, especially keeping it physically with you at all times. E.g. you could run Debian with LVM encryption and if you have a very secure password, no one's getting in there ever except yourself.