Storage, asset management and archive. Where does it all go?

Cameras

I’m guessing most people who know me or read Wide Open Camera’s blog were expecting that I would make a post on high-speed for me next entry.  That will come at a later time but today’s post is on something fairly important that most of us don’t take seriously until its too late.  What do I do with all my data?  Clearly videotape is dead to 95% of the market if not more and file based workflows are where we have been for the past 5 years and into the future.  So where do we put all that data?  We used to pop a tape or film-stock masters on the shelf (ideally in a climate controlled vault) and knew that a decade later we could pull footage and have confidence that our art (or crap) was safe.  That is not the case anymore.

Now most of us have had a hard drive fail by either the power supply crapping out or the needle parking on a platter.  Either way our content was gone and unless we paid $5000 for data recovery (which often is only partial) we lost it all.  Knowing this experience I still see many people using single disk or RAID 0 storage devices to keep their masters on and daily edits, etc.  Russian Roulette might sound like extreme words to use for this but basically if media is how you make a living and you are working with portable drives or basic RAID’s you are waiting to loose it all.  Your lively-hood, reputation, and mental stability is at stake.  You can’t expect a consumer hard drive to last more than a year.  Certain drives (which I will not name here) are good for approx 8 months before their power supplies fail during use, causing the read/write head to park on the platter.

Most of us understand that a good camera is worth every penny and most of us are lucky enough to recoup its value ten fold before its on to something new.  That same logic needs to be applied to our storage of the precious content we are making with said camera.  It makes no sense to spend $6K-$25K on a camera package, almost the same on a edit system, yet only $400-$1200 on storage with no archiving.  Its ironic that people will spend $600 on a P2 or SxS card but won’t spend more than $600 on a drive.  Mind you a proper RAID for video is going to cost you money as there is no cheating your way around it.  Ideally you want as many drives in a chassis as possible for speed and realistically you want some parity across the drives so a rebuild can happen when a failure has taken place.  RAID protection is smart and needed for what we do.  I sometimes see people stripping across several consumer / prosumer drives for speed in RAID 0 configuration.  This is probably the worst thing you could ever do because not only are you increasing the chances of loosing everything but you have no way of recovering your data.  This is a guaranteed failure waiting to happen.

During NAB whenever storage was mentioned the only thing I heard was about speed. Hearing comments like “I’m getting a thunderbolt drive so I can edit R3D files in realtime or I’m looking to forward to multiple streams of HD playback”, etc.  Not much mention of data protection, data integrity, archiving, etc.  Its really time to wake up and put real consideration into where and how we are storing our work.

Most of what I have been talking about here has been the defacto standard methods of small one or two man operations, but even more importantly is the small ad agencies popping up all over in the major markets.  I have seen and visited several of these smaller operations and am impressed with their editing platform choices, camera rentals or purchases, lighting, etc but not so much on the protection front.  Some of these agencies have one or two big clients which is a great launching pad for expansion, etc.  What I also notice in almost everyone of these budding agencies is a lack of archive and storage solutions that protect their Hero clients work.  Often it laid out to a stack of 1-2TB firewire or SATA drives that sit on the edit desk.  We vault and climate control our tapes and film stocks but I see these drives sitting on top of a hot macpro or near a window baking in the sun.  Not a great way to ensure longevity of a drive.   What these agencies and all of us for that matter need is a way to ensure that their clients work is safe.  There is nothing worse that having to explain to a client that their project is gone or a re-shoot is required.

One process I recommend to anyone who is doing real work for clients or even art for themselves is to master their data to an archive-able medium.  If you have an important project before you do anything you should back it up.  Not to a single spinning disk but rather to something more tangible like LTO-4/5 , XDCAM Optical or large multi-disk RAID (24 drives with RAID 5 or similar).  Backing up your movie, commercial, etc to LTO-5 is probably one of the smartest things you can do these days. They hold 1.5TB of data each and are index-able with metadata and will last over 50 years.  You can span projects and if used with an asset management system like CAT-DV or FLOW you can keep tabs on everything properly.  Asset management is a key part of file based workflows and only a handful of us are using it sadly.  What happens when you have 40 plus drives of projects and your Excel spreadsheet is corrupted?  How do you find what you have sitting on those drives? How do you know what is lost if one of those drives fails?  What record keeping system is in place to ensure everything is where it needs to be and can be found?  LTO-5 or a large archive RAID is good assurance but should also be coupled with asset management and proper meta-data tagging on ingest.  A proper asset management software package will also give you a thumbnail view of what you have and help facilitate shared storage workflows with your team.

Proper RAID storage, archive and asset management is expensive but a much needed component of the bigger picture.  File based workflows are not going away.  We will have more and more data even with more efficient codecs coming out every year.  One large RED project will have four or more times the amount of a say a XDCAM project’s assets.  Why do I mention this?  You will most likely loose more projects if your drive fails working with less data intense codecs.  You have more to lose and even more reason to back up everything.

The argument I hear most from clients and friends is that they do not want to invest (or their wording waste) their money on storage since it gets bigger and faster every year.  The same argument can be made with cameras.  They also will always have more features and cost less.  There is always something better down the line. Yes storage does get cheaper every year, but this is mainly in reference to consumer drives or smaller simple RAID’s.  Larger RAID solutions drop pricing marginally and over longer periods of time, but while adding more features and better protection.  You cannot afford to lose your data.

So whats a good solution for most of us that doesn’t cost $25k?  My advices would be to get an LTO-4 or LTO-5 and backup all your masters to that.  Keep your current workflow until you can afford (not that you can afford to lose your stuff) a proper RAID with protection.  This will at least give you piece of mind so that when your portable drive fails, you will always have your masters safe and sound on the shelf.  Ideally put your project file with these masters so you can at least rebuild your project and timeline when a failure happens.

We all need to start allocating at least 50% of our budgets to the other half of what we do.  Post.  Out of that 50% for post we need to allocate at least 75% of that to storage and archive.  You cannot afford not too.

Mike Sutton @MNS1974

Follow @MNS1974

P.S. to be clear I am biased about this subject and it is close to my heart.  I do work for an Film/HD/4K sales and rental company ( Rule Boston Camera ) that does sell shared storage, asset management and archive solutions and my girlfriend reps EditShare.  Don’t hold it against me. These are still best practices and you will thank me for it.

 

Jared Abrams
Jared Abrams is a cinematographer based in Hollywood, California. After many years as a professional camera assistant he switched over to still photography. About two years ago a new Canon camera changed the way the world sees both motion and still photography. He just happened to be in the right place at the right time.
  • http://twitter.com/videocataloger Robert Lönn

    Great article. I agree that some people take too lightly on the seriousness of the injury to the business a major disc crash can cause. 
    I think the argument should be turned around. It is not about cost, it is about what value you are protecting. Same way you protect your other valuables with insurance policy’s.

  • http://donnersens.org/ Philippe Kiener

    Good article, thanks. Would you have any LTO product that is a good deal ?

  • http://twitter.com/MNS1974 Michael N Sutton

     Agreed.  I have seen so many people loose their projects to $250 consumer drives that own $25,000 cameras.

  • http://twitter.com/jamesbenet James Benet

    Excellent post,  What about cloud storage as an extra redundancy step?  Amazon S3 or Google, others?   3TB on Amazon S3 is around $4500 US a year just for storage.

    There really should be a service that lets us backup redundantly on the cloud for a reasonable price. We rarely need to access the data and usually on a loss of data on our end so why does it cost so much? 

    If there was a reasonable price for “safe high capacity redundant storage”, I would be uploading now!

  • Dr Pixels

    Really great article, it is a shame that i didn’t take roper care of some of my drives and today one is failing bad and is all my family photo archive i so have it backup on other but the problem is i can’t tell if the backup is current and is about 200gb worth of 5 years of pictures. Now i have to recover file by file more than 40k pictures and videos. I did read before abot lto but can really know if thwy are for me or if i just build a pc with 6tb on a raid but really dont know which one to use for reliable and economical archiving. Any advise?

  • http://www.facebook.com/michelgallone Michel Gallone

    Just one comment: saying that LTOs will last 50 years is very optimistic. People were saying the same with AIT Tape based backup: guess what most tapes on which i backed up stuff in the late 90s are now unreadable. Same with DAT back up that I started in the late 80s early 90s. And both DAT and then AIT were considered the safest back up medias at the time.
    Well lets be honest:  with ALL Digital media there is absolutely no guaranty of long term storage.
    What Data Centers do though is to  transfer old data (on tapes or other medias) to newer medias every once in a while .
    So the same applies for us video professionals. If we want to make sure our data is still available in 50 years time, we  need to transfer to newer media every once in a while (every 5 years I assume).

  • http://www.alphapointtechnology.com/it-asset-management-software IT Asset Management

    Nice article. You’re right, a lot of people don’t want to invest in storage since the next best thing always comes out year after year. Can’t wait until we can store TBs of data in the cloud.

  • Pingback: Asset storage | Makulita