My father told me he wanted to make USB flash drives of all the scanned and digitized family photos and other assorted letters and mementos. He planned to distribute them to all family members hoping that at least one set would survive. When I explained that they ought to be recipes to new media every N number of years or risk deteriorating or becoming unreadable (like a floppy disk when you have no floppy drive), he was genuinely shocked. He lost interest in the project that he’d thought was so bullet proof.
To me, this is just another story of the music industry’s technical incompetence.
Even in the 1990s, everyone would have known that hard drives were not a long-term archival storage solution. This is like crumpling up a piece of paper, tossing it in the corner, then being upset decades later when your “archival solution” had issues.
The piece of paper is basically much more likely to survive in a corner barring a fire. I have crumpled pieces of paper from 20 years ago. My PATA hard drive… I don’t even have a computer with that connector
A bunch of paper tossed into a corner could get wet, mouldy, get munched on by rats, etc. But, I know what you mean. Spinning plates full of magnetized bits with a connector technology that only lasts a decade at most is hardly going to be reliable, even if stored under ideal conditions.
Not to mention bit rot
I’m sure they considered anyone telling them they needed to spend money to be a pain in the ass the same way companies don’t follow the recommendations of their IT departments.
The problem isn’t even the hard drives, it’s how they are managing them. There’s not many digital data storage solutions around that you can dump into a closet for a few decades and then still read.
You have to regularly test your hard drives, so that when one fails you can take your other copy of the data and put it on a new drive.
This is where “piracy” is actually the industry’s saving grace. Decades or centuries later, will record labels exist and be well-managed (and flush with cash) enough to preserve archival copies of their artists catalogs? Probably not.
Will obscure weirdos exist all around the world on Usenet, IRC, or seeding torrents? Possibly.
What is really being discussed here is archiving of master recordings and session files. The publically avaliable releases themselves aren’t really in jeopardy. Orthough piracy probably does provide an extra layer of security to more obscure releases.
Oh, fuck. Prince’s Vault… God I hope the estate has a plan to preserve all of that…
I thought I read somewhere that when they were making one of the Toy Story movies, there was some catastrophic data loss that nearly tanked the whole production. But then one of the animators came back from maternity and said wait, I think I have most of it synced to my home server? And the next thing you know, John Lasseter himself is barrelling down the highway to her place and it turned out yeah, she did have it.
The value of distributed redundancy
Something to be said for the wfh movement too.
Many films and tv programs survive only thanks to a total stranger keeping their own copy. For a long term survival of any media it has to be copied and distributed far and wide.
There is no “write and forget” solution. There never has been.
Do you think we have ORIGINALS or Greek or roman written texts? No, we have only those that have been copied over and over in the course of the centuries. Historians knows too well. And 90% of anything ever written by humans in all history has been lost, all that was written on more durable media than ours.
The future will hold only those memories of us that our descendants will take the time to copy over and over. Nothing that we will do today to preserve our media will last 1000 years in any case.
(Will we as a specie survive 1000 more years?)
Still, it our duty to preserve for the future as much as we can. If today’s historians are any guide, the most important bits will be those less valuable today: the ones nobody will care to actually preserve.
Citing Alessandro Barbero, a top notch Italian current historian, he would kill no know what a common passant had for breakfast in the tenth century. We know nothing about that, while we know a tiny little more about kings.
There is: mdiscs. Allegedly 1000 years durability even in Blu-ray format. Should be good enough for most important things. The best tapes AFAIK 30- 100 years
Problem is how to read the disk, especially after generations. Will they retain the knowledge to build and operate a device for this?
I wish there was a cheap and millennia-long lasting microfilm you could transfer books to. A projector is a pretty simple device to operate. Hmm that reminds me of “Last Words (2020)”.
Microfilms used to be sold as having a life expectancy of up 500 years. But in my experience they were a pain to use and the machines costly to maintain. The films would tear regularly too. Also the quality of the recorded image could be very poor sometimes.
Do you think we have ORIGINALS or Greek or roman written texts?
We do have originals of some much older texts, though (cuneiform on clay that was fired after impression seems to be a pretty good archival medium, overall). We’d probably have a lot more original Greek and Roman documents if they hadn’t been destroyed in wars and other disasters, or recycled for various purposes. There’s a big survival rate difference between documents that receive basic care throughout their lives—no rough handling handling, minimal direct sunlight exposure, and some degree of temperature and humidity control in the storage area—and those left to fend for themselves. That’s why old documents in surprisingly good condition sometimes turn up in caves, which tend to have constant temperature and humidity levels.
(But, yeah, current electronic media doesn’t have much chance, with select optical disk media stored under carefully chosen conditions offering the best chance for your files being retrievable decades later, if you can find a drive to read them on.)
BRB, etching all my MP3s into clay tablets
Sad little human. I have written my treatises into the warp and weft of reality itself. I have twisted my curiosity into the folds of your DNA and stamped my waxing madness into the ragged edges of the telomeres that mark your days as numbered. I have made of the stars a celestial QR code that burns across the skies of every planet, that burns across the eyes of every ape who stares into the night and asks “why?”. I announced The Work with a bang of gas and light and awe and set time itself into motion so my scripture could expand eternally into the infinite, benighted expanse.
Is this from something?
Sounds like an HFY prompt
I got four HDDs, some are almost 10 years old. They work great but I know that won’t be for forever.
Im really hoping, waiting, for a good dense long-term storage medium. It doesn’t have to be fast, but large, cheap, and durable. I want a way to backup my plex library, or even, daily backups of documents and project files, and I don’t want to think about them ever again.
Tape is cheap and durable if you store it properly. Except the tape drive is expensive af.
Microsoft is working on glass storage. A glass plate can last 10,000 years according to Microsoft. Hopefully that tech will get miniaturized and available to consumers within our lifetimes.
As a former audio engineer in the days where we still used it- tape can rot.
im not an audio engineer, but people didnt know this???
It seemed like the person I was talking to didn’t. The implication was that tape was viable as long-term storage. It isn’t. I’ve seen tapes rot after a year. DATs were especially prone to that, but even things like 2" multitrack audio tape can go bad that quickly.
M-Disk is rated to last like
1001000 years. They are also working on a 125 Terabyte CD. Optical storage is the way to go.Optical storage
so they’ve solved bitrot? nice
The main cause of bitrot in older disks is the organic dyes fading (aside from REALLY cheap disks where delamination was a problem), whereas M-Disc uses an inorganic carbon material
TIL, thanks!
Well yeah but BluRay is still much more expensive and smaller capacity. Lets hope this new 125TB disk works out
Punch cards? Stored correctly there’s no reason they couldn’t last many human lifetimes. But… Yeah it’ll take a while to encode everything.
I would have thought that with modern technology we could come up with something like punch cards but more space/time efficient. Physical storage of data - only one write cycle of course, but extremely durable. Even just the same system as punch cards but using tiny, tiny holes very close together on a roll of paper. Could be punched or read by a machine at high speed (compared to a regular punch card, presumably still Ber slow compared to flash media).
Paper doesn’t last centuries. Anyway, punched cards don’t have a storage density that’s adequate for modern data volumes. You need something that’ll durably store nanometre-sized marks.
Yes that’s what I’m thinking, some modernised physical data storage technique.
My system is to duplicate to fresh media once in a while. It’s more hands on, but it’s the only option I have. My NAS will be cloned to new drives in the next few years.
That’s the way to do it. Just keep bringing data forward even for media that sits in a drawer.
I explained that they ought to be recipes to new media every N number of years or risk deteriorating or becoming unreadable
This is important, and for some media, it should be more often than that.
People forget that flash memory uses electrical charge to store data. It’s not durable. If left unpowered for too long, that data will get corrupted. A failure might not even be visible without examining every bit of every file.
Keep backups. Include recovery data (e.g. PAR2). Store them on multiple media. Keep them well-maintained (e.g. give flash drives power). Mind their environment. Copy them to new storage devices before the old ones become obsolete.
It’s funny that with all our technology, paper is still the most durable storage medium (under normal conditions) that doesn’t cost an arm and a leg.
Yeah if you’re looking for long term it needs to be archival media. Many people think the flash drive will hold it forever but they are potentially the most fickle.
But what actually is “archival”?
Like, what technology normal person has access to counts at least as enthusiast level archival?
Magnetic tape, optical media, flash, HDD all rot away, potentially within frighteningly short timeframes and often with subtle bitrot.
Hard drives offer the best price/capacity ratio, but they need to be powered periodically (at least once or twice per year). As with any other storage medium, include parity data and have multiple backups to avoid data loss.
Tape is too expensive.
Optical media can also be pretty good as long as you get discs made from inorganic materials and store them properly. M-disc is supposed to last like 100 years. The biggest problem is that they are on the path to obsolescence and optical drives may stop being manufactured. Also, it’s a good idea to check on the condition of the discs periodically and redo any that shows signs of degradation (probably a good idea to replace non-M discs every 10 years regardless).
But regardless of the media, there is no archival method that doesn’t require active maintenance, like periodically checking the data, ensuring you have multiple backups, refreshing any aged media.
1960s style punch cards. Made of concrete.
I write these words in steel, for anything not set in metal cannot be trusted.
But it can be rusted.
M-DISC, at a guess. The media would last long enough at least for grandkids, who will have bigger things to worry about.
Don’t forget, you also need drives that work that long and connect to computers or some other device to utilize the bits, and the bus they use must be available and working, and the disk format they’re written in must be readable, and the images themselves encoded with an algorithm that we still have access to, etc. it’s not just the media.
I think it’s possible, thanks to the retro enthusiasts, we still have access to some things from the 70s and 80s, but they’re getting fewer and fewer, especially in a working state. That’s only 50yrs ago. What happens when you want to go 100? Or 500? A few thousand? We are familiar with journals from the Civil War, and have found items and notes from Egypt, Roman, and Ancient Greek civilizations, how can we preserve what happened in the currently information rich time we live in, for future generations? Especially as much of it migrates online to blog posts and social networks and news sites that eventually shut down due to corporate issues or shifting internet traffic?
Upload it to the cloud and make it someone else’s problem to deal with keeping up with the physical medium changes. Then your descendants only have to worry about figuring out how to deal with an outdated file format they can no longer open… and even when they can finally open it, it’d be super low quality… just like how we have to squint really hard at videos from VCDs now days.
There have been plenty of cloud services that have shut down and taken their data offline. And plenty of current ones deleted data after users have gone inactive. Or require constant payments to keep accounts active. Cloud, as it exits now, is not the answer to the archival question.
You’ll be very hard pressed to find anything else that’d out last the day when all three of AWS, Azure and GCP shutdown and take their data offline.
I get it though, Lemmy doesn’t want to admit these services exist other than to dunk on them in the most anti-corporate fashion… so continue to pretend such is the case!
They take your data down pretty quick when you die and stop paying for it. And as much as we all want to think AWS and GCP and Azure are sticking around forever there’s no reason at this time to believe they will be around in 100+ years.
Like, what technology normal person has access to counts at least as enthusiast level archival?
Cloud storage? Store it on 2 different providers like B2 and iDrive or something, pretty low complexity.
Is it? It’s rather expensive and would you really know, if the data is gone or corrupted?
You’d have to download every single file in certain intervals and check it. That’s not really low complexity.
Good backup software will just do that as part of its thing, like Restic for example.
And who does that?
I think you don’t really get my point. I’m not arguing that there are no ways to archive data. I’m arguing that there are no technologies available for average Joe.
It is hardly a good strategy to basically set up half a datacenter at home.
Probably M Disk as others have said for consumer use, but Microsoft is working on storing data in glass that could last for potentially 100,000 years +. Not that you’d ever likely have that in your home. Although, maybe by 2100 we will, who knows. https://www.microsoft.com/en-us/research/project/project-silica/
This is really the only way. Ironically not far off clay tabs just much much smaller.
USB-A is best bet today, will live longer than other formats and USB adaptors will still exist when USB-A will disappear entirely.
Huh? USB is a connector, not an archival format.
Werent we talking about usb flash drives?
Since usb flash drives use usb, I think we can keep using them to store data in long term, rather than using floppy, cd or other analogic archive.The problem is that USB flash drives don’t keep their data intact for very long when they’re powered down. It lasts long enough for everyday use, but not even as long as a hard disk for archival.
when they’re powered down.
There’s no periodic cell refresh in flash memory like there is in DRAM. When USB sticks are plugged in, all you are doing is powering up the flash chip and interface ICs.
You’d have to read a block then write it back to actually refresh the stored charges in the cells.
Do SSDs do that automatically in the background, or is all the data I’m not actively refreshing gradually rotting away?
These people haven’t heard of zfs then?
Zfs is just software raid, not an archival /backup solution. Sure, you can hold data on a zfs array for long term, but not without active maintenance (powering the drives periodically, replacing old drives, doing some kind of data refresh / scrubbing) and backups.
That’s why I back up my data on stone tablets in Cunieform.
Seriously though, if you wanted data to last for centuries, what would be your best bet? Would it be some sort of 3D-printed mechanical storage? At least plastics are generally not biodegradable, though they are photodegradable, so I guess you’d want to stick your archive in a dry cave somewhere?
Or what about this idea of encoding the data in the DNA of some microbe and cutting it loose? What could possibly go wrong?
Please stop that, DNA is good for mutations, not for long-term consistency.
Well I guess I’m picturing DNA encoding like a RAID billion in terms of redundancy, so with some checksumming, you ought be able to sort out any mutations? But I’m no geneticist.