On the Internet you will find:
- A gazillion websites and weblogs describing the functionality of tools and software e.g. many sites under http://populair.eu/software or “the top 100 free utils to…” so you go ahead and install all of the things you need and think are cool and handy.
- A gazillion fragmented handy need-to-know blogpostings, forum threads and utils for managing e.g. “backups of your WordPress MySQL” or “backup of Firefox” which you can all read one by one.
However you would want one generic approach of managing ALL that software and not one specific asset. You will often read that “backups” need to take place and how cool that specific backup utility is. It , however, never describes what to backup in the first place. You will also find out that installation is cool but managing the installation is not so cool and which is made invisible in the costs in time for you to be able to use it. You need to take care of backups (what to backup?) of knowing a way to restore your previous instance (how and what to restore), manage the installation (if you faced an issue will you remember the workaround(s) the next time?) and you will need to take care of updates since they often fix bugs or even fix important security issues (how to keep track of updates?), often you will need to record license keys or files (how to keep track of it) and a lot of other operational aspects which become only clear when things go wrong.
You can try to go to each of the sites of the “top 100 free programs to…” and look at the information provided: you will find out where to download it, the cool features, sometimes how to install it but pretty much never an “operations” tab which actually tells you how to maintain it e.g. “what to backup” or “how to restore”. This is information you will need to search for in forums. Weird isn’t it? While you would expect a “functional usage” versus “operational usage” split on these pages where operations include installation, backup and restore.
I thought I would share with you some things I have setup for managing my home software environment. I’m still thinking on how to do some things so it’s by no means a final solution for managing my home software environment but maybe it will give you some tips. I’m using both Windows and Linux machines in my household.
In essense it is not about having “that specific cool backup utility” but defining a process within your household for managing the operational environment. That is what needed in the end.
Without such a process there will be these rare panic peeks or constant reliance on “the smart nephew who knows about computers” every time that something is wrong.
1. Give all assets a unique Id
I started by giving all assets an id. Assets could be either software, backups or both.
In the image on the left you can see part of the wiki page showing the assets by ID e.g. 0001 = firefox (and extensions) which consists of both the software and its backups or 0023 = bookmarks which are merely backups of my older bookmarks from a variety of sources without any software associated to it.
These Id’s are then used all throughout the further story. It gets you focused on each asset.
An asset can be only a program e.g. “VLC” or a program with specific configuration data e.g. “total commander” or a program with specific configuration data and user data e.g. “outlook” or just only data e.g. “mysql backup” or “linkedin contacts”, as long as it something specific it gets its own id.
2. Describe the assets in the household wiki
I then described each asset on its own page on the wiki. E.g. I have chapters on : download locations, my own repository, a copy of the license key / license type e.g. gnu, how to install it, what to backup, where to back it up and other things needed to know around the asset as well as where to get support (forums, wikis, weblogs) and so on.
You can regard this as both the backup plan and the disaster recovery plan and the license management etc… which you may find in larger companies but within my household all in one simple page combined.
It will make sure I can very quickly recover to a working environment again in case of problems and it makes sure I will have a backup of everything needed. It also makes me think of things and provides me with a page with interesting links/discussions for each asset.
While writing this I noticed that a lot of information is undocumented e.g. just try for yourself finding all the things you need to backup for your belastingdienst programma’s of the past years.
3. Create a local software repository “\\nas\software\essentials”
On my NAS I have created a share \software which contains a LOT of different programs, development tools, scripts, etc… etc… I have accumulated things there from the past 20 years which means you can click through there for years (try a download of ftp://ftp.boulder.ibm.com to get a feeling for this and then add the Microsoft MSDN kit and a full download of http://sourceforge.net/ and a large part of the items mentioned in each of the sites on http://populair.eu/software) and the svn directories of e.g. the WordPress plugins and customizations etc… etc..
So I have created a new directory in the \software share named “essential”. Within this directory I use the same numbering scheme as defined.
In each of these directories I keep my software. It has also some challenges e.g. try for yourself locating all the plugins you use for Firefox or locating all the extensions in XPI format or even .exe format. And after that try to find out how you can restore the plugins after a system crash and how Firefox actually knows that “adobe reader” is to being used in all profile instances.
It has the advantage that I can quickly re-install the software when a system crash occurs or pick individual items to install in new environments. It has the disadvantage that new updates of software keeps coming out so it is always outdated and in a home environment there is simply no time to keep checking all updates on all software
4. Create a backup repository “\\nas\backup\essentials”
Of course I then have a share named \backups on which I save my backups.
I use the same numbering scheme so that e.g. my profiles of firefox (and profiles.ini) are saved under the same number “0001”.
I have different kind of folders:
- I have created subfolders of the machine name inside the main folder if I have different installations on different machines
- I have created only one main backup folder if the content is the same independent of the machine the software is being used on
- I have created only one main backup folder if the content is to being merged from different machines e.g. the password data from roboform which gets updated from different machines or in other words: I want to have access to the latest updated password collection from each machine.
So these folders do not only contain backup data of local installed applications it also has e.g. the backup MySQL data of all my web applications such as this site they all get their unique number.
5. Create backup scripts
Regarding of the backup software you use, you still need to know WHAT to backup.
So I have for each machine a backup script with this information which can have different forms or use different tools e.g. remote backups of a sql db is different than a local xcopy of some files or a copy of my linux config data.
It can be hard to find out all the different pieces needed for a backup, underneath you get a feeling for my approach on it.
6. Rename Windows “installed Software” descriptions
Not many people know that you can rename the descriptions of the installed software list on a Windows Machine.
But you can just right-mouse click a description and then rename the description.
So I gave the items the correct ID.
The big advantage: It will give you slowly an overview of the software that you haven’t thought about yet since that software will not have a number yet.
Just take a look under start > settings > control panel > Add or Remove Programs to give yourself an indication of it.
7. procedure for installations on Windows machines
On each Windows machine I try to install all applications under the same application directory “c:\edward\app”.
I use again the same numbering scheme for the installed software
the name “edward” is badly choosen since now, afterwards, i realize that it may not be handy for some other machines. I will change that in the future.
In each of these folders there is only the installed software.
I use a different directory than the default “c:\program files” because I want to distinquish between what’s default installed on it and what I added myself (not all machines are fully under my control).
8. procedure for data on Windows machines
On each Windows machine I have all the data under the same data directory.
I use again the same numbering scheme for all installed software.
For example the firefox profiles are in the firefox data directory and the Total Commander ini files, plugins, custom iconbar are in the total commander data directory while my nokia phone backups are under the nokia data directory.
Not all applications allow this configuration and it is pretty hard to find out all the spots where data is saved. Under Windows there is many times data under the
c:\Documents and Settings\Administrator\Application Data\ or
c:\Documents and Settings\Administrator\Local Settings\Application Data\ or
in the windows registry
So to be clear it is this data directory which (among others) gets backed up regularly to the NAS \backup repository and NOT anything from the \edward\app\ directory, since that is a specific installation for a specific machine.
9. Some other tips
1. Developers of software not always document all of this, so it is handy to use software like sandboxie to find out all the different places where the application nestles itself before finding the correct backup procedure (and recovery!) (though there is a long list of other utilities helping with this). It also takes some time to get to know an application to auto configure it for a different data directory so scanning forums and mailing lists of particular software is many times needed because sometimes tweaks are needed.
2. Document everything, especially issues and things you had to google for yourself because in a year time you will have forgotten everything again. I use a central wiki for all of this information.
3. It is wise to have close by the complete list of registration keys and passwords used (e.g. for the different e-mail accounts, vmware servers, ftp accounts) etc… so in case of recovery that you have everything in the same spot. Just be careful enough to encrypt this information and store it in a safe place. I use the same numbering scheme for all of this.
4. One of the tricks I use for tagging the folders I have “processed” or “under control” is by giving them a different icon. E.g. within the c:\Documents and Settings\Administrator\Application Data\ directory I simply give each of the folders that I have already processed a different icon instead of the default yellow icon, so that I know what is under control and what not.
5. You can also use the numbering scheme for cloud assets so e.g. websites where you have e.g. contact data or other data you want to keep. You can then describe in the wiki whats there and how to use it e.g. photo’s on flickr or contacts in linkedin (which you can sync with e.g. an outlook plugin).
6. You have to really try deleting the app and then restoring to find out if it works. E.g. for firefox you might find that by placing the profiles back in another location that you will also need to edit the “extension.ini” file with updated locations of the new root dir of the profiles (or else your extension will no longer work).
I will follow up in some future blogpostings on specific procedures for specific software.
10. An example: 0003 Total Commander
1. the wiki pages describe Total Commander, the website, the website of the support forum, download locations and installing and configuring the tool, it also describes the registry entries it uses and what needs to be backed up. It also provides me with my personal license key and the different plugins I use are described.
It describes that the following registry key could be run to set the installation versus data directories for both the local machine and the specific user which overrides the local machine settings.
It also describes that within wincmd.ini, the configuration file, certain directories should be changed to reflect the data directory e.g.
Secure FTP Connections=c:\edward\data\0003_tc\plugins\wfx\sftp\plugin_sftp.wfx
It also describes that within some specific plugins some specific configuration files need to be edited to reflect the data directory e.g. within the “icons bar configuration file” :
2. installation takes place in the app dir
3. the data dir contains all data which is configuration data like settings and plugins used. It can be discussed whether plugins should be part of it. But in the case of TC and e.g. Firefox it is much easier to consider them as part of the configuration data than as seperate programs. This is because they are part of the configuration set.
4. Backup dir
In the backup dir I then simply copy this data directory once and again, in this case I have made subfolders for each different machine because I have different ini files on each different machine.
REM 0003 Total Commander Backup
xcopy "c:\edward\data\tc\*.*" "R:\0003_tc\L3KHG13-1951CZ1\" /E /V /H /Y
xcopy "c:\edward\app\tc\wincmd.key" "R:\0003_tc\L3KHG13-1951CZ1\" /E /V /H /Y
11. An idea: Household Software Operations wiki
For many (web) applications it takes a long time to find out how they work, which registry keys they use, finding some kind of support forum or newsgroup out there, how to split program and data, on which strange places it actually stores data, which files should be edited when restoring to different locations, and so on and so on.
Within (larger) companies there is often a dedicated department (including helpdesk) which takes care of these things. They use specific process frameworks like ITIL and use specific software for managing this environment. It contains a lot of different fields of expertise and a lot of different domains. Within a household there is just uhm… 1 person who has to somehow micro scale these operations to a (lucky) much simpler environment but in essence there are still the same tasks.
Within a household we don’t get any education on the things we use, so we scroll through sites, weblogs, faqs, forums and newsgroups for both operational knowledge as well as functional knowledge. For additional tooling needed we do the same. It provides quite some Google traffic for all of us doing the same thing over and over again.
Since I haven’t found it yet I want to start, whenever I have time left, a wiki for household operations. It would enable households to quickly assemble the information needed for managing their software environment and prevent them for spending a lot of time searching and assembling this information for themselves.
I will place it under http://www.householdframework.org/