View Full Version : How Do I Safely Store My Data..Long Term
I have a volume of 3.6TB that is just about full. It is backed up to a network raid 5 volume of much larger size and to yet another local 5TB raid 5 Time Machine. I would like to offload older photos and videos.
25GB Blu-Ray media is now less than one U.S. dollar, but is such a backup a reliable way to go? I've had multiple Blu-Ray media cease to be readable within a year.
Tim Gray
09-16-2011, 07:29
If you aren't using tape drives, I think the easiest, most affordable method of backup is to do what you are doing. Multiple copies on multiple drives. Store a copy in a different location. I don't think you have any choice but to migrate your data every few years to newer and larger hard drives.
kshapero
09-16-2011, 07:36
I use Carbonite and a 2 Terabyte back up drive.
Long term... how long? 10 years, 20 years? 200 years? 2000 years ? That are the questions here.
All storage drives will eventually fail. Some after 3 years, some after 20 years, of sitting in the dust ... Multiple copies, multiple storage locations help. Constantly keeping up/repairing the infrastructure helps. So buy a new 10++ Terabyte storage device every 2 years, load everything onto the new one, all that forever stuff, and try to keep a dozen + older ones alive until they slowly die. Then you will always have a few usable devices.
Unless a big power surge, a Texas wildfire, a flood wipes out the whole lot ... So set up sets of yearly renewing super big storage devices in various locations around the globe. At grandmothers, the cottage in Colorado, ... until granny dies, the house burns, gets sold ... a nuclear disaster occurs, a flood, ...
But all else will eventually fail.
Why do you want to keep stuff for long, forever?
willie_901
09-16-2011, 14:28
The basic plan is:
redundant on-site storage (which you seem to have covered)
off-site storage at a safe location on a regular schedule
The definition of a safe location is debatable. Clearly a bank safety deposit box is safer than the closet of a friend/relative on the next block. The safety deposit box is much less convenient. Decades ago we sent scientific data (monthly backups) on tape to an abandoned salt mine. This is really safe, but really inconvenient.
Another possibility is the Cloud. But 3.6 TB on the cloud will be rather expensive.
porktaco
09-16-2011, 14:34
this works for me
"DATA! SIT DATA! STAY... STAYYYYY..... STAY!"
ColSebastianMoran
09-16-2011, 15:17
I've given up on CD/DVD media. Not that I've had so many errors, but life is too short to try to figure out which disc for a certain photo.
I think plain old hard disk drives are the way to go. I keep my files on a big external drive, and there are two other copies of this same drive. I retire the drives after two years and copy everything to the next new bigger drive. Current files are also backed up by Apple's Time Machine.
File format is another question. So much for PhotoCD as a long term archival approach. Who will be able to read a current camera raw in twenty years. Or a Lightroom catalog? I do keep Nikon raw (NEF), xmp sidecar files, and a good jpg of the final rendered image.
deerstalkr
09-16-2011, 19:11
I store recent photos, or 'keepers' on an external hard drive and on my current machine. It's a lot more convenient for me to access them, but I do not depend on them for long term storage.
I archive photos on a DVD disc and have all images archived and stored appropriately; the Lightroom catalogue is cross-referenced with a separate database in Excel. Image location is noted, and so are disc locations. It's not infallible., but it works well for me.
Funny, a few years ago I found negatives from 1979-93.
I was trying to use DVDs but heard that they fail. I have a few external HDs now. They're getting progressively larger and I'm up to 4T.
Darthfeeble
09-16-2011, 19:29
Gad's! I'm a piker with a mere 300 gigs. I had a bunch of DVD media fail on me and did some research on archival media (CD & DVD). Seems most of the grocery store media is aluminum based and will oxidize if not sealed good. Gold based media has a hundred year rating with the expectation of longer according to the sellers and several reviews I read. This particular brand was touted as the best. http://www.google.com/search?hl=en&client=safari&rls=en&q=taiyo+yuden+archival+dvd&revid=1159918373&stick=H4sIAAAAAAAAAONgEuJJLErOyCxLzFFIKUuRYlDiDknM rMxXiCxNSc1TYlBiAAAiV2D7IwAAAA&sa=X&ei=RRN0TpPPI_HdiALX14C0Ag&ved=0CFwQ9AsoAA
I heard that pros were making copies of their best printed digital images using Kodachrome film.
Gad's! I'm a piker with a mere 300 gigs. I had a bunch of DVD media fail on me and did some research on archival media (CD & DVD). Seems most of the grocery store media is aluminum based and will oxidize if not sealed good. Gold based media has a hundred year rating with the expectation of longer according to the sellers and several reviews I read. This particular brand was touted as the best. http://www.google.com/search?hl=en&client=safari&rls=en&q=taiyo+yuden+archival+dvd&revid=1159918373&stick=H4sIAAAAAAAAAONgEuJJLErOyCxLzFFIKUuRYlDiDknM rMxXiCxNSc1TYlBiAAAiV2D7IwAAAA&sa=X&ei=RRN0TpPPI_HdiALX14C0Ag&ved=0CFwQ9AsoAA
DVDs are a non-starter. The capacity is quite small. Are there Blu-Ray equivalents at less than their weight in gold?
Harry Lime
09-16-2011, 20:17
Shoot film. Preferably black and white.
I have had a number of four bay raid units and have always avoided Drobo because of their slow interface, but just purchased my first... there are now rebates on many of their units. They are amazingly forgiving. You can use different size drives and continue to change drive size, one at a time, until you reach drive capacity. I think my four bay tops out at 17TB. Now have seven TB of drives which results in a bit less than 5TB of space. Really great choice.
Bob Michaels
09-16-2011, 20:46
My $.02 for FWIW:
I am no Luddite. I have 40 years in the management of technology companies. I have seen much storage technology obsoleted over the years. I like to think my experience causes me to be pragmatic and broad minded.
I believe anyone who is debating HDD vs. gold plated DVDs is missing the forest for focusing on the trees.
I archive my important photos by acid free prints in a light tight box. I am not going to be impacted by technological obsolesce. Yes, they are on multiple HDDs as well, but I remember the 8" floppy disk.
You say you have 3.6Tb of photos. Allow me to be gracious and ask if all of that is really worth saving? Is there a possibility you could never find the good photo you are looking for because it is included with so much else?
I would encourage you to let history and some very broad minded thinking influence what you consider to be viable long term storage.
Snowbuzz
09-16-2011, 21:53
Excellent post, Bob.
I'm printing to a Blurb album once or twice per year. Not exactly exhibition quality, but better than rarely viewing photos on a hard drive. Just sent off another 947 images. I've been recently rating my photos, but have another 25K to go. So, the idea of backing up just the best photos makes sense. My videos are the time consuming items and I just don't have time to edit them, but imagine that someday I will. Probably a fantasy that will never come true. I need to get a handle on the video side of things.
Richard G
09-17-2011, 03:22
What Bob said. Print. And negatives. For digital files, print will last longest. That's a good idea to photograph keeper digital pictures on whatever will be the stable film successor to Kodachrome. Meanwhile, editing to a hard drive of keepers is a good idea, with minimum two copies. And you need to access those drives regularly to make sure the files are OK. And when one file is corrupted, reformat and copy after checking the other two, and so on. Maybe checking the file structure of the whole disk would go some way towards this, but I doubt that would be certain enough.
I have every negative of every film photograph I have taken with the exception of four rolls, and that's since 1972. My neat little portable LaCie hard drive is the old FireWire connection, just five years old, and I have one expensive FireWire 400 to FireWire 800 cable, that works, for now. Print....print....print....stop Dave,.....will you stop Dave?
Richard G
09-17-2011, 03:27
How long do you NEED to keep pictures? Probably only long enough for your children to acquire a spouse, who'll be interested, and maybe not even that long. But for their children to then see their parents as children, that's a lot of fun. What's the chance my non-photographer friends will have a single working digital file of their infant children 20 years later? I suspect it will be close to zero.
Negatives/ CDs, flickr and external hard drive storage for all images for me.
I just had another HD (and 1TB at that) go south, suddenly and with no warning. Luckily I had backed up images only the week before but lost other historical data. And it is not the first and won't be the last. I used to backup to DVD but it has become such a pain and from what I hear, no guarantee. So it's multiple HD's.
But I still have all the negs I have shot since 1983. Last week I took only film on a shoot for the first time in many years in protest of the failed drive. And it felt good :) These negs will be around as least as long. Doesn't help when I'm shooting digital at 8fps at my son's soccer match though.
I like to also do a 'blurb' book of my travels which is another useable backup of sorts.
John
BTW.. I have all of my film stored since 1966. Today, much of my digital storage is actually of film images. Oh.. and I know right where to find the images from 1966 onward.
I heard that pros were making copies of their best printed digital images using Kodachrome film.
Kodachrome seems a bit problematic due to the high contrast.
I have one tried to put my hands on Ilfochrome, which theoretically may be the optimal solution for long term archival storage of color photography.
It is more stable that Kodachrome, and suited to reproduction of both transparency and prints. It even seems easy to process at home.
It was impossible though. Only large corporates or museums buy the stuff in batch from ilford.
details here:
http://www.microcolour.com/mci03.htm
and once you reach the conclusion that it's perfect for the job, try to get some...:bang::bang:
djonesii
09-17-2011, 05:06
On a monthly basis, my favorite family images are sent up to the cloud, but only at web size .... This is bad.
Once a year, a book is sent up to Blurb, and I have a printed copy. If they go under, this is bad for a long term archive plan, but the printed copy will not self destruct. :eek:
I have two full sets of hard drives, on internal, and one external, over a decade, I'm up to just over 2 terra. Just bought 2 new 3 terra drives, and they will allow me to take the two old 1 terra drives to have a third off site set. As some have mentioned, even with a careful naming scheme, I still have trouble finding photos. With lightroom, I really need to start tagging, and this will help. I have been on the ever increasing M-Pix/Gigabyte merry go round, and see no way off it!
I scan then dispose of all my negatives, I was trying to archive them in a physical world, but it's just not worth it for me.
YMMV.
Dave
dave lackey
09-17-2011, 05:29
This is one of those threads that constantly amazes and amuses me for so many reasons. :angel:
Upon my own demise, I would only hope that a handful of worthy prints, negatives and digital images will pass on to my grandchildren. The other tens of thousands of images...meh.;)
dave lackey
09-17-2011, 05:33
I just had another HD (and 1TB at that) go south, suddenly and with no warning. Luckily I had backed up images only the week before but lost other historical data. And it is not the first and won't be the last. I used to backup to DVD but it has become such a pain and from what I hear, no guarantee. So it's multiple HD's.
But I still have all the negs I have shot since 1983. Last week I took only film on a shoot for the first time in many years in protest of the failed drive. And it felt good :) These negs will be around as least as long. Doesn't help when I'm shooting digital at 8fps at my son's soccer match though.
I like to also do a 'blurb' book of my travels which is another useable backup of sorts.
John
Oh, CD's and DVD's not reliable? How about confirmation on that rumor?:angel:
8fps...Nikon F5? Dang, wish I still had mine even though I don't do sports photography anymore.:)
As far as books, it is probably the very best way to assure that at least some of your work survives a long time with the negatives...just place them in a fireproof safe and distribute a lot of books and prints.
Shoot film. Preferably black and white.
On polyester based film.
Richard G
09-17-2011, 05:42
Mike Johnson covered demise in a good post on his Online Photographer site. For a few good pictures not to be thrown out with thousands of rubbish ones you will have to edit out a small cache before you die.
Well I'm keeping all mine ... so the will have to take a few trips to the dump when I go ;)
Much of the 3.6TB (Aperture Vault 463GB) is video. If I had more time I could edit that down. Of course, this is like items in my garage.. what is the point in keeping it if you don't view it with some regularity? I actually know the answer to that. When my wife wants to see the kids when they were just learning to walk, talk, play sports.. or fill-in the blanks, I want to be able to show her what she wants to see. In other words, I would like to stay married. I wonder if my father has any old 8mm and if he does, what condition and how would he view them? Somewhat the same problem.
Brian Sweeney
09-17-2011, 15:46
My 8mm, Super-8, and Sound Super-8 films are in great condition. I found an Ektasound movie projector at a yard sale for $1, and an 8mm projector at a thrift store for $10. My original projectors wore out, a Eumig/Fairchild 711R and a K-Mart Dual 8.
Finding anything among 3Tb of data, even if you could look through it, will be the daunting task of your heirs. If it's too much for you to edit down, they might just toss the whole thing.
My Kodachromes and Cibachromes from the sixtys still look good.:D
To the people who keep insisting 'shoot film' I wish you luck.
I've done a couple of archival scanning jobs from negatives dating back to the early part of last century and some of them were serioulsy on their last legs. Admittedly storage had been pretty casual but a negative is not a permanent storage solution for a photographic image. It may be so within your own life time and that may be enough for you ... but for future generations film is just a material on it's way back to mother earth!
Oh, CD's and DVD's not reliable? How about confirmation on that rumor?:angel:
I can't say anything definite but... I've got a lot of burned CD's from perhaps 5 or 6 years ago (not photos however).
Today half of them are unreadable or when opened have no data on them even though looking at the CD you can see something has been burned. They used to work when I opened them so I'm pretty upset that I cannot access the files any longer. The only explanation I can give to why no data can be read from them is either time or the technology is outdated (though they are just simple CD's so i'd say it's unlikely).
But on the other hand, other CD's work. Music CD's always seem to work, so too do movie DVD's that I've bought years ago.
So I can't explain it, but the fact is that something has happened to those CD's that were sitting safely in storage and what was on them is lost forever. There's always a risk of storing anything. The best bet is backing up in multiple locations.
Originally, running shoes came in them. Now they hold hundreds and hundreds of my negs and transparencies. Nothing digital matches their longevity.
To the people who keep insisting 'shoot film' I wish you luck.
I've done a couple of archival scanning jobs from negatives dating back to the early part of last century and some of them were serioulsy on their last legs. Admittedly storage had been pretty casual but a negative is not a permanent storage solution for a photographic image. It may be so within your own life time and that may be enough for you ... but for future generations film is just a material on it's way back to mother earth!
Keith,
How do you think would have printed pictures saved on a hard drive of the early part of the 20th century kept in a pretty casual storage?:rolleyes:
Film is a chemical support and as such has a limited lifetime, even if well taken care of, but this lifetime can be very long for B+W material.
It is also more sensitive to physical destruction since you have one original.
On the other hand, digital support is very sensitive, and if not properly backed up every few years, the data will eventually disappear. And it's not a slow fading. Without full continuity in the back up process, you loose everything. think about it: If I die and my kids are at a point where they are not aware of the problem, and do not backup properly my pictures. Even if they suddenly re-value the digital family archive ten years after, it's most likely to be too late.
I remember that what brought me to photography was a couple of shoe boxes with hundred of family pictures, some dating form the late 18th century, that was just put in a closet at my grandmother's for decades.
Try that with a HD/CD/DVD...
Digital and chemical (B+W) can both be maintained for long enough. The dangers are different that can harm them and both present some advantages over the other.
If I was a pro photographer backed with a large agency that I know will take care of my pictures, I would think digital is the best. For my small home operation, meanwhile, I choose film, and invest a great deal in trying to keep them safe.
I can't say anything definite but... I've got a lot of burned CD's from perhaps 5 or 6 years ago (not photos however).
Today half of them are unreadable or when opened have no data on them even though looking at the CD you can see something has been burned. They used to work when I opened them so I'm pretty upset that I cannot access the files any longer. The only explanation I can give to why no data can be read from them is either time or the technology is outdated (though they are just simple CD's so i'd say it's unlikely).
But on the other hand, other CD's work. Music CD's always seem to work, so too do movie DVD's that I've bought years ago.
So I can't explain it, but the fact is that something has happened to those CD's that were sitting safely in storage and what was on them is lost forever. There's always a risk of storing anything. The best bet is backing up in multiple locations.
Ari,
Music CD's, movie DVD's sold to you are printed in a press. Unless they are heavily scratched, you can read them because the metal supporting the data is fine in the plastic.
On the contrary, when you burn a blank CD, you use a laser to alter a support that is sensitive to light, and remains so after the burning. That is why we are advised to store optical data in the dark, and apparently that's also why you better buy good media.
Exposure to light slowly alters the metal layer in a burnt CD, and the sensivity of the support may even make it degrade (although slower) in darkness.
If you want to use safely CD/DVD technology, buy the best support possible (it was TY a couple of years ago), burn at least TWO copies. write down the date of burning, and I suggest reburning new copies every five years or so. Of course, dark storage...
Keith,
How do you think would have printed pictures saved on a hard drive of the early part of the 20th century kept in a pretty casual storage?:rolleyes:
Film is a chemical support and as such has a limited lifetime, even if well taken care of, but this lifetime can be very long for B+W material.
It is also more sensitive to physical destruction since you have one original.
On the other hand, digital support is very sensitive, and if not properly backed up every few years, the data will eventually disappear. And it's not a slow fading. Without full continuity in the back up process, you loose everything. think about it: If I die and my kids are at a point where they are not aware of the problem, and do not backup properly my pictures. Even if they suddenly re-value the digital family archive ten years after, it's most likely to be too late.
I remember that what brought me to photography was a couple of shoe boxes with hundred of family pictures, some dating form the late 18th century, that was just put in a closet at my grandmother's for decades.
Try that with a HD/CD/DVD...
Digital and chemical (B+W) can both be maintained for long enough. The dangers are different that can harm them and both present some advantages over the other.
If I was a pro photographer backed with a large agency that I know will take care of my pictures, I would think digital is the best. For my small home operation, meanwhile, I choose film, and invest a great deal in trying to keep them safe.
I'm not saying film isn't effective to a point but by the laws of nature it has to degrade over time ... whatever that time period may be I'm not qualified to say.
A digital file can be duplicated when necessary and transfered to fresh storage media ... but of course here I'm making the assumption that the technology to do so and the ability to read the file remains accessible in the future ... which is pretty optomistic I guess. :D
There is no real solution I suspect and like film the jpeg, tif or whatever may be long gone and we'll be left wondering what what we should have done!
Freakscene
09-18-2011, 22:48
From someone who really knows: http://leica-users.org/NYLUG-2011.pdf
http://en.wikipedia.org/wiki/Brian_Reid_(computer_scientist)
Marty
My earliest files are dated from Feb 1981: they're online, backed up, and also reside on the original 9-track tape. It's doubtful the average Joe has that kind of dedication to digital preservation. My recommendation to those that want to try is to migrate your entire stash every year or so to a different disk (just to keep in practice), and to get properly versed in magnetic tape (to avoid correlated error in the hard drive industry). Have you seen an entire hard drive product line collapse due to systematic error? Not pretty. Needless to say, you also need offsite copies, documentation, and absolutely standard formats for the volume archive and image file.
Because you don't have any of these tedious routines with a stack of toned B&W printed, or KR25 (RIP), there's real value to some backup in analog form. :) Plus, you don't need advanced training to retrieve and enjoy the pics (relatives will thank you).
I would say that a RAID 5 (or 6) is your best option. I am not sure about the longevity of Blue Ray discs and burning that volume of data would cost lot of time. If you want additional backup to you RAID system that maybe consider to make additional back up of selected data (smaller volume)
I am using just a simple RAID 1 but I am still below 1 TB ...
Bob Michaels
09-19-2011, 04:02
From someone who really knows: http://leica-users.org/NYLUG-2011.pdf
Marty: thanks. He illustrates the point much better than I ever could.
I recommend viewing by anyone who thinks any electronic storage is viable long term.
Texsport
10-31-2011, 16:02
Shoot film. Preferably black and white.
That is the best solution. Now you have to figure out how to store all those slides.
Texsport
wgerrard
10-31-2011, 19:52
Decide what you want to protect and then back it up in more than one location and, if you can, more than one format.
I engage in a bit of overkill, perhaps, because I had the hardware. I use Aperture. Its library lives on my laptop. It works with images on a 2-tb external drive (referenced masters, in the parlance). Original out-of-camera files live on another external drive. "Keeper" Jpegs live on the 2-tb drive. Everything is backed up to a third external drive. Original files are also backed up online to an Amazon S3 bucket, and the Jpegs are backed up to a different online service. I use Flickr and have a website, so anything there can be retrieved if necessary.
Sync your raid to a cloud based storage...
RObert Budding
11-01-2011, 02:19
I've been looking at FreeNAS because it supports the ZFS file system.
http://www.freenas.org/
Pherdinand
11-01-2011, 03:04
print them all on paper ;)
djonesii
11-01-2011, 04:11
Finding anything among 3Tb of data, even if you could look through it, will be the daunting task of your heirs. If it's too much for you to edit down, they might just toss the whole thing.
My wife's grand mother passed away. Her husband had many cameras, one actually made it to me. Thus, he had many negatives.
I got a box of them. ...... 5 books of 100 sleeves, and about 10 folders/envelopes with around another 200 negatives. Most are 6x4.5, some are 6x9, some seem to be cut to 6X6, and a few cut to oddball sizes ...
There is no meta data at all, no organization except loosely by date/location as the negatives tend to be grouped.
Some of the negatives has deteriorated, others have not, none have been a total loss.
Assuming that my spindled medium does not die, and Adobe meta data is still readable, my heirs will have a much simpler job!
That said, it will be incumbent on me to keep the medium current until my death!
Dave
That is not surprising. The negatives were on nitrate-based film stock. An unstable material and subject to spontaneous combustion. Even current acetate "safety" film will deteriorate over time. Not polyester.
To the people who keep insisting 'shoot film' I wish you luck.
I've done a couple of archival scanning jobs from negatives dating back to the early part of last century and some of them were serioulsy on their last legs. Admittedly storage had been pretty casual but a negative is not a permanent storage solution for a photographic image. It may be so within your own life time and that may be enough for you ... but for future generations film is just a material on it's way back to mother earth!
Jamie Pillers
11-08-2011, 10:34
I just read about a new DVD-type archival photo storage technology called "M Discs". They're made by a company called Millenniata. The technology is discussed in the latest issue of Shutterbug magazine. Here's the link to the company: http://millenniata.com/m-disc/
They claim that the the M Discs are made of inert materials that don't degrade over time. Apparently current style DVD's lose their ability to hold onto data due to the degradation of organic dye material used in their construction.
Data can be written on the new discs with most modern DVD recording technology. They are write-only. Once data is put on them, they can't be erased or written over.
If what I read is true, this could be the big breakthrough in banishing our concerns about archival-ness of image files! :)
vBulletin® v3.6.8, Copyright ©2000-2013, Jelsoft Enterprises Ltd.