Carnelian high-res works scans are 525.9Mb but the low-res samples are 19.4Mb! Scary.
What influenced you to pick a .ZIP over a .RAR?
(Should I duck?)
(Should I duck?)
I think zip is a good choice. It's very standard and works everywhere.
Great idea! I've always used DownThemAll previously.
Great idea! I've always used DownThemAll previously.
ZIP has a clear spec that I can use to implement it. As far as I know, RAR has none at all.
I've always found 7zip better at BMP, but i haven't got any source or knowledge about whether it's better for jpeg or png yet.
But not like PNG is a killer anyway. Most of us have pretty good broadband connections.
But not like PNG is a killer anyway. Most of us have pretty good broadband connections.
I get the impression it's not to further compress already compressed images (png, jpg), just to make it easier to download. They're likely just stored in the zip file rather than compressed.aoie_emesai said:
I've always found 7zip better at BMP, but i haven't got any source or knowledge about whether it's better for jpeg or png yet.
But not like PNG is a killer anyway. Most of us have pretty good broadband connections.
Some technical details, for anyone interested:
The ZIP processing is implemented as a Lighttpd module. ZIPs aren't precreated; the contents are sent directly from the individual files. This saves a lot of I/O (some of the pools are quite large). It's able to use Lighttpd's AIO directly, so it's just as fast as downloading the files manually. (A ZIP with uncompressed files is just the files themselves with some data stuffed around each file.)
It also makes mirroring easy; generally when you open a ZIP link, it's going to create the ZIP (actually, a small control file that tells Lighttpd what to put in the ZIP), copies that over to the mirror, and redirects you there.
It handles resuming cleanly: when you open the link, you're redirected to a timestamped version of the ZIP at the time you opened it, so if you use a download manager and it resumes (or even pauses, waits a day, and then resumes), it'll keep giving you the same version of the file and not corrupt your download. (This depends on your download manager being smart enough to remember the redirect and not reload the original URL when resuming or unpausing. If yours doesn't do this, get a new one!) This is also why there's a small delay the first time someone opens a ZIP link when there's no current control file; it has to copy the new one to the mirror, which takes a second or two.
This does mean that the link you're redirected to has to expire eventually, since we have no real way of knowing when your client is done with it. (I actually havn't implemented expiry yet--the files are like 2k each--but it'll probably be something like a week or two after you first open the link.)
This also means it's important that if you link to the file externally, that you use the real link on the pool page and not the one you're redirected to. (Keeping in mind that, currently, you need to be logged in anyway. I'd prefer people just link to the pool itself and not the ZIP.)
Finally, zip64 extensions aren't currently supported. We're not at the point where we have 5-gig pools...
The ZIP processing is implemented as a Lighttpd module. ZIPs aren't precreated; the contents are sent directly from the individual files. This saves a lot of I/O (some of the pools are quite large). It's able to use Lighttpd's AIO directly, so it's just as fast as downloading the files manually. (A ZIP with uncompressed files is just the files themselves with some data stuffed around each file.)
It also makes mirroring easy; generally when you open a ZIP link, it's going to create the ZIP (actually, a small control file that tells Lighttpd what to put in the ZIP), copies that over to the mirror, and redirects you there.
It handles resuming cleanly: when you open the link, you're redirected to a timestamped version of the ZIP at the time you opened it, so if you use a download manager and it resumes (or even pauses, waits a day, and then resumes), it'll keep giving you the same version of the file and not corrupt your download. (This depends on your download manager being smart enough to remember the redirect and not reload the original URL when resuming or unpausing. If yours doesn't do this, get a new one!) This is also why there's a small delay the first time someone opens a ZIP link when there's no current control file; it has to copy the new one to the mirror, which takes a second or two.
This does mean that the link you're redirected to has to expire eventually, since we have no real way of knowing when your client is done with it. (I actually havn't implemented expiry yet--the files are like 2k each--but it'll probably be something like a week or two after you first open the link.)
This also means it's important that if you link to the file externally, that you use the real link on the pool page and not the one you're redirected to. (Keeping in mind that, currently, you need to be logged in anyway. I'd prefer people just link to the pool itself and not the ZIP.)
Finally, zip64 extensions aren't currently supported. We're not at the point where we have 5-gig pools...
Neat, so it's just like tar in respect to being able to stream data into the container. I didn't know you could do that with zip.
That's one of my favorite reason for file compression. Corrupted data will not decompress properly giving you a second chance to see what's the issue with the file.
Even though this happens to me about 5%-10% of the time, i just force it to decompress it for me with confirmation ^^
...Though I read a few articles on file compression efficiency, the binaries got me. My knowledge don't go past that >.<
Don't think my math classes have helped me much...
Even though this happens to me about 5%-10% of the time, i just force it to decompress it for me with confirmation ^^
...Though I read a few articles on file compression efficiency, the binaries got me. My knowledge don't go past that >.<
Don't think my math classes have helped me much...
petopeto
Pool ZIPs
One reason is that they usually want a complete archive of the whole release to be available, which hasn't been possible here. I've added a download link to the pool page, which allows downloading the whole pool without having to use a download manager. A ZIP of the sample images is also available. Filenames are based on the pool order (this may be expanded).
(There are other reasons, another big one being that it's too hard to set up a working mass-uploader. One step at a time...)
Currently, you need to be logged in to use it (external links to the ZIP link will redirect to the pool page). I may expand this to allow hotlinks to go directly to the ZIP for people who are already logged in. (I just don't want people linking to the ZIPs blind--we're not a file hosting site, after all.)
This is intended to allow downloading of real pools, not for people to create their own personal archives to download. Don't do that (it'll make a mess of the pools and you'll end up with numbered filenames); use something like DownThemAll.