Storage, asset management and archive. Where does it all go?
I’m guessing most people who know me or read Wide Open Camera’s blog were expecting that I would make a post on high-speed for me next entry. That will come at a later time but today’s post is on something fairly important that most of us don’t take seriously until its too late. What do I do with all my data? Clearly videotape is dead to 95% of the market if not more and file based workflows are where we have been for the past 5 years and into the future. So where do we put all that data? We used to pop a tape or film-stock masters on the shelf (ideally in a climate controlled vault) and knew that a decade later we could pull footage and have confidence that our art (or crap) was safe. That is not the case anymore.
Now most of us have had a hard drive fail by either the power supply crapping out or the needle parking on a platter. Either way our content was gone and unless we paid $5000 for data recovery (which often is only partial) we lost it all. Knowing this experience I still see many people using single disk or RAID 0 storage devices to keep their masters on and daily edits, etc. Russian Roulette might sound like extreme words to use for this but basically if media is how you make a living and you are working with portable drives or basic RAID’s you are waiting to loose it all. Your lively-hood, reputation, and mental stability is at stake. You can’t expect a consumer hard drive to last more than a year. Certain drives (which I will not name here) are good for approx 8 months before their power supplies fail during use, causing the read/write head to park on the platter.
Most of us understand that a good camera is worth every penny and most of us are lucky enough to recoup its value ten fold before its on to something new. That same logic needs to be applied to our storage of the precious content we are making with said camera. It makes no sense to spend $6K-$25K on a camera package, almost the same on a edit system, yet only $400-$1200 on storage with no archiving. Its ironic that people will spend $600 on a P2 or SxS card but won’t spend more than $600 on a drive. Mind you a proper RAID for video is going to cost you money as there is no cheating your way around it. Ideally you want as many drives in a chassis as possible for speed and realistically you want some parity across the drives so a rebuild can happen when a failure has taken place. RAID protection is smart and needed for what we do. I sometimes see people stripping across several consumer / prosumer drives for speed in RAID 0 configuration. This is probably the worst thing you could ever do because not only are you increasing the chances of loosing everything but you have no way of recovering your data. This is a guaranteed failure waiting to happen.
During NAB whenever storage was mentioned the only thing I heard was about speed. Hearing comments like “I’m getting a thunderbolt drive so I can edit R3D files in realtime or I’m looking to forward to multiple streams of HD playback”, etc. Not much mention of data protection, data integrity, archiving, etc. Its really time to wake up and put real consideration into where and how we are storing our work.
Most of what I have been talking about here has been the defacto standard methods of small one or two man operations, but even more importantly is the small ad agencies popping up all over in the major markets. I have seen and visited several of these smaller operations and am impressed with their editing platform choices, camera rentals or purchases, lighting, etc but not so much on the protection front. Some of these agencies have one or two big clients which is a great launching pad for expansion, etc. What I also notice in almost everyone of these budding agencies is a lack of archive and storage solutions that protect their Hero clients work. Often it laid out to a stack of 1-2TB firewire or SATA drives that sit on the edit desk. We vault and climate control our tapes and film stocks but I see these drives sitting on top of a hot macpro or near a window baking in the sun. Not a great way to ensure longevity of a drive. What these agencies and all of us for that matter need is a way to ensure that their clients work is safe. There is nothing worse that having to explain to a client that their project is gone or a re-shoot is required.
One process I recommend to anyone who is doing real work for clients or even art for themselves is to master their data to an archive-able medium. If you have an important project before you do anything you should back it up. Not to a single spinning disk but rather to something more tangible like LTO-4/5 , XDCAM Optical or large multi-disk RAID (24 drives with RAID 5 or similar). Backing up your movie, commercial, etc to LTO-5 is probably one of the smartest things you can do these days. They hold 1.5TB of data each and are index-able with metadata and will last over 50 years. You can span projects and if used with an asset management system like CAT-DV or FLOW you can keep tabs on everything properly. Asset management is a key part of file based workflows and only a handful of us are using it sadly. What happens when you have 40 plus drives of projects and your Excel spreadsheet is corrupted? How do you find what you have sitting on those drives? How do you know what is lost if one of those drives fails? What record keeping system is in place to ensure everything is where it needs to be and can be found? LTO-5 or a large archive RAID is good assurance but should also be coupled with asset management and proper meta-data tagging on ingest. A proper asset management software package will also give you a thumbnail view of what you have and help facilitate shared storage workflows with your team.
Proper RAID storage, archive and asset management is expensive but a much needed component of the bigger picture. File based workflows are not going away. We will have more and more data even with more efficient codecs coming out every year. One large RED project will have four or more times the amount of a say a XDCAM project’s assets. Why do I mention this? You will most likely loose more projects if your drive fails working with less data intense codecs. You have more to lose and even more reason to back up everything.
The argument I hear most from clients and friends is that they do not want to invest (or their wording waste) their money on storage since it gets bigger and faster every year. The same argument can be made with cameras. They also will always have more features and cost less. There is always something better down the line. Yes storage does get cheaper every year, but this is mainly in reference to consumer drives or smaller simple RAID’s. Larger RAID solutions drop pricing marginally and over longer periods of time, but while adding more features and better protection. You cannot afford to lose your data.
So whats a good solution for most of us that doesn’t cost $25k? My advices would be to get an LTO-4 or LTO-5 and backup all your masters to that. Keep your current workflow until you can afford (not that you can afford to lose your stuff) a proper RAID with protection. This will at least give you piece of mind so that when your portable drive fails, you will always have your masters safe and sound on the shelf. Ideally put your project file with these masters so you can at least rebuild your project and timeline when a failure happens.
We all need to start allocating at least 50% of our budgets to the other half of what we do. Post. Out of that 50% for post we need to allocate at least 75% of that to storage and archive. You cannot afford not too.
Mike Sutton @MNS1974
P.S. to be clear I am biased about this subject and it is close to my heart. I do work for an Film/HD/4K sales and rental company ( Rule Boston Camera ) that does sell shared storage, asset management and archive solutions and my girlfriend reps EditShare. Don’t hold it against me. These are still best practices and you will thank me for it.
Related Posts
New Product
Pingback: Asset storage | Makulita