Tag Archives: backup

Oneindige opslag voor $49 per jaar bij Bitcasa doen of niet doen?

February 18, 2013

imageIk blogde al eerder over Bitcasa, waar je oneindige schijfruimte krijgt voor $99 per jaar. (en een special price: $49 voor eerste gebruikers voor het eerste jaar). Ik ben nu aan het twijfelen of ik gebruik ga maken van deze aanbieding:

  1. De software werkt goed. Ik heb deze nu de afgelopen tijd gebruikt en ik moet zeggen: al mijn gigabytes blijven netjes bewaard, sterker nog, ik kan ook nog eens oudere versies terughalen. Dus qua file opslag is het ideaal en het werkt vrij ok en snel. In tegenstelling tot sommige berichten op het net heb ik zelf geen last van traagheid.
  2. In tegenstelling tot Dropbox zijn de infinite folders ideaal: je pleurt er alles in en de storage is niet je lokale schijf maar  ergens in de cloud. Je moet dus wel altijd connectie hebben, maar thuis heb ik dat ook vrijwel altijd. Onderweg kan ik altijd een paar copietjes maken.
  3. De send to optie is ook ideaal, gelijk aan you-send-it kan ik grote bestanden doormailen of per twitter of facebook sharen.
  4. Het zou betekenen dat alles wat ik op de NAS heb staan wat puur “files” zijn (en dus niet mijn within-household web applicaties) naar Bitcasa kan sturen en het betekent dat ik me nooit meer zorgen hoef te maken over storage problemen. In tien jaar tijd is $99 * 10  = $ 1000 dollar dat is pakweg 750 euro. Gelet op het feit dat ik ook lokaal in RAID moet draaien en schijven moet backuppen is dat absoluut goedkoop. goedkoper dan een volledige NAS met schijven (maar let op het gaat over file storage niet over applicaties runnen op een server) (daar is een NAS wel handig voor).

Continue reading

How to organize your home Software Environment

April 5, 2009

On the Internet you will find:

  • A gazillion websites and weblogs describing the functionality of tools and software e.g. many sites under http://populair.eu/software or “the top 100 free utils to…” so you go ahead and install all of the things you need and think are cool and handy.
  • A gazillion fragmented handy need-to-know blogpostings, forum threads and utils for managing e.g. “backups of your WordPress MySQL” or “backup of Firefox” which you can all read one by one.

However you would want one generic approach of managing ALL that software and not one specific asset. You will often read that “backups” need to take place and how cool that specific backup utility is. It , however, never describes what to backup in the first place. You will also find out that installation is cool but managing the installation is not so cool and which is made invisible in the costs in time for you to be able to use it. You need to take care of backups (what to backup?) of knowing a way to restore your previous instance (how and what to restore), manage the installation (if you faced an issue will you remember the workaround(s) the next time?) and you will need to take care of updates since they often fix bugs or even fix important security issues (how to keep track of updates?), often you will need to record license keys or files (how to keep track of it) and a lot of other operational aspects which become only clear when things go wrong.

You can try to go to each of the sites of the “top 100 free programs to…” and look at the information provided: you will find out where to download it, the cool features, sometimes how to install it but pretty much never an “operations” tab which actually tells you how to maintain it e.g. “what to backup” or “how to restore”. This is information you will need to search for in forums. Weird isn’t it? While you would expect a “functional usage” versus “operational usage” split on these pages where operations include installation, backup and restore.

I thought I would share with you some things I have setup for managing my home software environment. I’m still thinking on how to do some things so it’s by no means a final solution for managing my home software environment but maybe it will give you some tips. I’m using both Windows and Linux machines in my household.

In essense it is not about having “that specific cool backup utility” but defining a process within your household for managing the operational environment. That is what needed in the end.

Without such a process there will be these rare panic peeks or constant reliance on “the smart nephew who knows about computers” every time that something is wrong.

image1. Give all assets a unique Id

I started by giving all assets an id. Assets could be either software, backups or both.

In the image on the left you can see part of the wiki page showing the assets by ID e.g. 0001 = firefox (and extensions) which consists of both the software and its backups or 0023 = bookmarks which are merely backups of my older bookmarks from a variety of sources without any software associated to it.

These Id’s are then used all throughout the further story. It gets you focused on each asset.

An asset can be only a program e.g. “VLC” or a program with specific configuration data e.g. “total commander” or a program with specific configuration data and user data e.g. “outlook” or just only data e.g. “mysql backup” or “linkedin contacts”, as long as it something specific it gets its own id.


image2. Describe the assets in the household wiki

I then described each asset on its own page on the wiki. E.g. I have chapters on : download locations, my own repository, a copy of the license key / license type e.g. gnu, how to install it, what to backup, where to back it up and other things needed to know around the asset as well as where to get support (forums, wikis, weblogs) and so on.

You can regard this as both the backup plan and the disaster recovery plan and the license management etc… which you may find in larger companies but within my household all in one simple page combined.

It will make sure I can very quickly recover to a working environment again in case of problems and it makes sure I will have a backup of everything needed. It also makes me think of things and provides me with a page with interesting links/discussions for each asset.

While writing this I noticed that a lot of information is undocumented e.g. just try for yourself finding all the things you need to backup for your belastingdienst programma’s of the past years.


image3. Create a local software repository “\\nas\software\essentials”

On my NAS I have created a share \software which contains a LOT of different programs, development tools, scripts, etc… etc… I have accumulated things there from the past 20 years which means you can click through there for years (try a download of ftp://ftp.boulder.ibm.com to get a feeling for this and then add the Microsoft MSDN kit and a full download of http://sourceforge.net/ and a large part of the items mentioned in each of the sites on http://populair.eu/software) and the svn directories of e.g. the WordPress plugins and customizations etc… etc..

So I have created a new directory in the \software share named “essential”. Within this directory I use the same numbering scheme as defined.

In each of these directories I keep my software. It has also some challenges e.g. try for yourself locating all the plugins you use for Firefox or locating all the extensions in XPI format or even .exe format. And  after that try to find out how you can restore the plugins after a system crash and how Firefox actually knows that “adobe reader” is to being used in all profile instances.

It has the advantage that I can quickly re-install the software when a system crash occurs or pick individual items to install in new environments. It has the disadvantage that new updates of software keeps coming out so it is always outdated and in a home environment there is simply no time to keep checking all updates on all software


image 4. Create a backup repository “\\nas\backup\essentials”

Of course I then have a share named \backups on which I save my backups.

I use the same numbering scheme so that e.g. my profiles of firefox (and profiles.ini) are saved under the same number “0001”.

I have different kind of folders:

- I have created subfolders of the machine name inside the main folder if I have different installations on different machines
- I have created only one main backup folder if the content is the same independent of the machine the software is being used on
- I have created only one main backup folder if the content is to being merged from different machines e.g. the password data from roboform which gets updated from different machines or in other words: I want to have access to the latest updated password collection from each machine.

So these folders do not only contain  backup data of local installed applications it also has e.g. the backup MySQL data of all my web applications such as this site they all get their unique number.


image 5. Create backup scripts

Regarding of the backup software you use, you still need to know WHAT to backup.

So I have for each machine a backup script with this information which can have different forms or use different tools e.g. remote backups of a sql db is different than a local xcopy of some files or a copy of my linux config data.

It can be hard to find out all the different pieces needed for a backup, underneath you get a feeling for my approach on it.


image 6. Rename Windows “installed Software” descriptions

Not many people know that you can rename the descriptions of the installed software list on a Windows Machine.

But you can just right-mouse click a description and then rename the description.

So I gave the items the correct ID.

The big advantage: It will give you slowly an overview of the software that you haven’t thought about yet since that software will not have a number yet.

Just take a look under start >  settings > control panel > Add or Remove Programs to give yourself an indication of it.


image 7. procedure for installations on Windows machines

On each Windows machine I try to install all applications under the same application directory “c:\edward\app”.

I use again the same numbering scheme for the installed software

the name “edward” is badly choosen since now, afterwards, i realize that it may not be handy for some other machines. I will change that in the future.

In each of these folders there is only the installed software.

I use a different directory than the default “c:\program files” because I want to distinquish between what’s default installed on it and what I added myself (not all machines are fully under my control).


image

8. procedure for data on Windows machines

On each Windows machine I have all the data under the same data directory.

I use again the same numbering scheme for all installed software.

For example the firefox profiles are in the firefox data directory and the Total Commander ini files, plugins, custom iconbar are in the total commander data directory while my nokia phone backups are under the nokia data directory.

Not all applications allow this configuration and it is pretty hard to find out all the spots where data is saved. Under Windows there is many times data under the

c:\Documents and Settings\Administrator\Application Data\ or
c:\Documents and Settings\Administrator\Local Settings\Application Data\ or
in the windows registry

So to be clear it is this data directory which (among others) gets backed up regularly to the NAS \backup repository and NOT anything from the \edward\app\ directory, since that is a specific installation for a specific machine.


9. Some other tips

1. Developers of software not always document all of this, so it is handy to use software like sandboxie to find out all the different places where the application nestles itself before finding the correct backup procedure (and recovery!) (though there is a long list of other utilities helping with this). It also takes some time to get to know an application to auto configure it for a different data directory so scanning forums and mailing lists of particular software is many times needed because sometimes tweaks are needed.

2. Document everything, especially issues and things you had to google for yourself because in a year time you will have forgotten everything again. I use a central wiki for all of this information.

3. It is wise to have close by the complete list of registration keys and passwords used (e.g. for the different e-mail accounts, vmware servers, ftp accounts) etc… so in case of recovery that you have everything in the same spot. Just be careful enough to encrypt this information and store it in a safe place. I use the same numbering scheme for all of this.

image 4. One of the tricks I use for tagging the folders I have “processed” or “under control” is by giving them a different icon. E.g. within the c:\Documents and Settings\Administrator\Application Data\ directory I simply give each of the folders that I have already processed a different icon instead of the default yellow icon, so that I know what is under control and what not.

 

5. You can also use the numbering scheme for cloud assets so e.g. websites where you have e.g. contact data or other data you want to keep. You can then describe in the wiki whats there and how to use it e.g. photo’s on flickr or contacts in linkedin (which you can sync with e.g. an outlook plugin).

6. You have to really try deleting the app and then restoring to find out if it works. E.g. for firefox you might find that by placing the profiles back in another location that you will also need to edit the “extension.ini” file with updated locations of the new root dir of the profiles (or else your extension will no longer work).

I will follow up in some future blogpostings on specific procedures for specific software.


10. An example: 0003 Total Commander

1. the wiki pages describe Total Commander, the website, the website of the support forum, download locations and installing and configuring the tool, it also describes the registry entries it uses and what needs to be backed up. It also provides me with my personal license key and the different plugins I use are described.

It describes that the following registry key could be run to set the installation versus data directories for both the local machine and the specific user which overrides the local machine settings.

[HKEY_LOCAL_MACHINE\SOFTWARE\Ghisler]
 
[HKEY_LOCAL_MACHINE\SOFTWARE\Ghisler\Total Commander]
"IniFileName"="C:\\Edward\\data\\0003_tc\\wincmd.ini"
"FtpIniName"="C:\\Edward\\data\\0003_tc\\wcx_ftp.ini"
"InstallDir"="C:\\Edward\\data\\0003_tc"
 
[HKEY_CURRENT_USER\Software\Ghisler]
 
[HKEY_CURRENT_USER\Software\Ghisler\Total Commander]
"IniFileName"="C:\\Edward\\data\\0003_tc\\wincmd.ini"
"FtpIniName"="C:\\Edward\\data\\0003_tc\\wcx_ftp.ini"
"InstallDir"="C:\\Edward\\app\\0003_tc"

It also describes that within wincmd.ini, the configuration file, certain directories should be changed to reflect the data directory e.g.

Secure FTP Connections=c:\edward\data\0003_tc\plugins\wfx\sftp\plugin_sftp.wfx
WinCE Device=c:\edward\data\0003_tc\plugins\wfx\wincefs\wincefs.wfx
HTTP Browser=c:\edward\data\0003_tc\plugins\wfx\httpbrowser\httpbrowser.wfx

It also describes that within some specific plugins some specific configuration files need to be edited to reflect the data directory e.g. within the “icons bar configuration file” :

button3=c:\edward\data\0003_tc\icons\Icons\X-Qute.dll,11
cmd3=cm_GotoPreviousDir
button4=c:\edward\data\0003_tc\icons\Icons\X-Qute.dll,12
cmd4=cm_GotoNextDir

2. installation takes place in the app dir

image

3. the data dir contains all data which is configuration data like settings and plugins used. It can be discussed whether plugins should be part of it. But in the case of TC and e.g. Firefox it is much easier to consider them as part of the configuration data than as seperate programs. This is because they are part of the configuration set.

image

4. Backup dir

In the backup dir I then simply copy this data directory once and again, in this case I have made subfolders for each different machine because I have different ini files on each different machine.

image

REM 0003 Total Commander Backup
xcopy "c:\edward\data\tc\*.*" "R:\0003_tc\L3KHG13-1951CZ1\" /E /V /H /Y
xcopy "c:\edward\app\tc\wincmd.key" "R:\0003_tc\L3KHG13-1951CZ1\" /E /V /H /Y


11. An idea: Household Software Operations wiki

For many (web) applications it takes a long time to find out how they work, which registry keys they use, finding some kind of support forum or newsgroup out there, how to split program and data, on which strange places it actually stores data, which files should be edited when restoring to different locations, and so on and so on.

Within (larger) companies there is often a dedicated department (including helpdesk) which takes care of these things. They use specific process frameworks like ITIL and use specific software for managing this environment. It contains a lot of different fields of expertise and a lot of different domains. Within a household there is just uhm… 1 person who has to somehow micro scale these operations to a (lucky) much simpler environment but in essence there are still the same tasks.

Within a household we don’t get any education on the things we use, so we scroll through sites, weblogs, faqs, forums and newsgroups for both operational knowledge as well as functional knowledge. For additional tooling needed we do the same. It provides quite some Google traffic for all of us doing the same thing over and over again.

Since I haven’t found it yet I want to start, whenever I have time left, a wiki for household operations. It would enable households to quickly assemble the information needed for managing their software environment and prevent them for spending a lot of time searching and assembling this information for themselves.

I will place it under http://www.householdframework.org/

Backup and operations of your PC or Laptop

April 3, 2009

After I lost 2 months of mail of Outlook I started thinking on the root causes of the problem on why I didn’t push the button to backup and it brought me to a wider problem/gap in the household IT landscape.

Within most enterprises there is a complete large department “operations”, they operate according to process frameworks like ITIL and they use advanced software to govern the installed software base, backups, security and much much more.

Of course one household is much too small for such an overkill on tooling and processes but still what we can learn is that apparently there is a gap between total chaos and full control since a household still is faced with doing these backups and having to do a restore once in a while and in need of operational information.

The reason that most households live in chaos around backups, security and operations of their “IT environment” is that the knowledge is not out there. It’s simply for some strange reason not on the Internet. Households are left totally on their own.

If you Google or surf the Internet for software you will find a gazillion websites which will show you the FUNCTIONAL description of software and how cool it is to use it. E.g. most all sites on http://populair.eu/software. but for MANAGING your software you have to go out and try to gather stuff from forums, discussions lists, faqs, and so on.

Let’s take “firefox” for example: you will find a gazillion websites and weblogs on what kind of cool things you can do with Firefox. When you look for backups you will need to actively go and look for a product specific for Firefox (e.g. the profiles information on the wikipedia help pages) which will do “backups” (e.g. a well known extension/add-on). However when you want a backup solution for your household you do not want to install for all the dozens of pieces of software seperate backups tools. Since you do not want to run or even spend the time and energy to find backup products for all you single pieces of software. You want something simple, just 1 backup tool that finds you stuff and can also handle disaster recovery and recovery of older version of your installed base.

Furthermore knowledge on what to backup is scattered or sometimes not even out there. E.g. what do you actually need to backup from Firefox? (profiles directories, profile.ini, more?) or from Total Commander (ini files, key, extensions)  or from Active Sync? or from Google Earth? or from Outlook? It will take you a long time investigating each product or even sandboxing each product and go out hunting for registry keys, configuration files, data on so on.

This does not stand on itself. Households are left on their own when it comes down to “a way to manage all those zillions of passwords”, “a way to manage licenses and keys”, “a way to govern security”, “finding out yourself how to create a how network”, “creating a central software repository”, “finding out what to backup”, “finding out how to and what to restore”. Everyone is doing the same thing for themselves over and over again and shared at the coffee machine in individual bits and pieces. For the less computer minded people: they call up their “technical friend/nephew/collegeau) to help them with operational aspects.

So I have started with some basic ideas on the governance on this. I make an “application directory” (to distinquish it from the /program files directory for practical needs) and an “data dir”, which should contain the bulk of stuff to backup. I also began giving out id’s to the things that are installed in most cases and to describe against those numbers, where the installation software is located (e.g. firefox but also each extension and theme and plugin) and how backup and recovery is handled. I also started writing a batch file which works according to these numbers so I can see what it backups (in some cases a merge/sync is needed instead of a backup). Against those same numbers I have added chapters in our wiki.

In time I will put this information under an “operational” chapter on the household framework (http://www.householdframework.org). It’s really missing out there.

Lost all my Outlook mail!

March 29, 2009

I was playing with firefox to set up some more profiles (I’m coding some more auto generation for my startpages directly from the firefox bookmarks)(e.g. http://populair.eu is fully auto-generated from my firefox bookmarks).

However when I accidently created a firefox profile directly in the root of my c:\edward directory, I decided to undo it, just "delete profile", "check", "next". *krrrrrrrrr* *realization* *stop, break,stop process*

But — it already was halfway deleting my complete c:\edward directory which obviously contains everything I want to backup and save. The bad thing is that my last manual backup was on the 1st of februari, meaning: I lost the last 2 months. Most of it, as far as I can see is recoverable (can’t really determine the exact listing in it) (probably some more will turn up)

But… the c:\edward\outlook directory is unrecovarable meaning… I lost the last 2 months of e-mails, which contained an AWFUL lot of todo e-mails, request for webpages, other directories full of needed saved attachments, additions and the latest agenda appointments.

*ARGH* *ARGH* *ARGH*

The not smart thing is that I have an auto-backup tool INSTALLED that auto copies this directory to my NAS. However… I didnt find the time yet in the past two months to manually hit that single button. It’s a good move anyway to make that one obligatory now (wise lesson again).

So…. IF you have sent my an e-mail lately and I haven’t replied (I also import all gmail ones) then … please send again. I also lost my agenda so if I miss an appointment thats why.

update: I found: Bug 302087 – Inadequate warning before Profile Manager deletes non-Mozilla files