how do passwords work?

newbie here. so consider this paradox: let's say you have a pass protected rar file. now the password should be stored somewhere in the file to compare against the inputted password. so if it's there how come you cant just extract it?

>>1
Encryption.

so consider this paradox
Why does no-on on /prog/ ever use the word paradox correctly

I don't know anything about RAR encryption beyond the fact that it uses AES, but I'm assuming it just tries to decrypt with the password the user provides and errors out when the first few blocks don't form a legitimate RAR header. That would make a lot more sense than storing the password in the file, which would be a retarded idea..

>>1
Alternatively you can store the cryptographic hash of the password, which would be a bitch to reverse.

>>5
You could store it, encrypted.

>>4
The header is in the clear. Password mismatch is generally assumed when the checksums differ.

>>6
mind=blown

>>7
a checksum is really just a hash, so yeah, encryption.

>>7
There's certainly a cleartext header at the start of an encrypted RAR file, but do you have a citation for the presence of any checksum or any reason to believe the payload isn't just a straightforward encrypted RAR file? It's annoyingly hard to find good information on the topic (mainly thanks to idiots like >>1 who are just looking for crackers), and I don't particularly want to spend my evening poring over a clusterfuck of Sepples code.

I cannot believe that you responded to this.

>>10
I don't have references, just experience. An example is when I enter the wrong password my local unrar doesn't know for sure what's wrong, complaining of a bad CRC and suggesting that since it needs a password, perhaps I got it wrong. Otherwise bad CRC is always reported as a corrupt archive. I'd say that's fairly conclusive.

I know that the CRCs are not part of the payload (this would be pretty fucking stupid if you think about it.) However, there are two ways to produce rar files (encrypted or not), one is pkzip-style (a collection of compressed files) and the other is tgz-style (a compressed collection of files--"solid" compression.) It's common to use solid compression on the unencrypted format, and the other form on encrypted files (from observation.) I do believe these are the defaults for virtually every rar compressor, though I have found at least one encrypted rar with solid compression. With the solid encrypted format, from observation, there is still a CRC in the clear. I expect there are individual CRCs to be found in a subheader as well, but I really didn't check. I do believe there is a "TOC"/subheader is encrypted in this case, but I don't remember this part clearly.

I do believe there is a "TOC"/subheader which is encrypted in this case,

>>12
From your description it really sounds more as if it's trying to decrypt the payload directly and then compare the results with a CRC in the header, and not at all storing a hash of the password anywhere. So >>9 is wrong.

>>14
No, there's certainly no password stored in the file (and there's really no need, since checksums are desirable even without encryption.) That is almost never the road taken to password protected files. The old utilities tended to use XOR without CRCs, but they could be cracked quickly (since XOR is fast) and easily by a substring search, unless the key is really long--like OTP.

But you could look at it as analogous to one-way password encryption in the sense that you have to match the CRC with the decrypted file. This analogy quickly falls apart when you investigate the nuances though.

ITT fucking idiots who don't know that:

1. The RAR file format (not compression algorithm) has official documentation

2. There are two opensource (not really "free", you RMS-obsessed neckbreads) implementations able to extract all versions of the format, including any combination of solid archiving and encryption

3. The defaults of all shipping RAR versions (there's only one RAR compressor) have been not solid and not encrypt headers ("file names" in UI and docs). Anything else has been explicitly specified by the user

The UnRAR source is fucking trivial to read, it took me literally 10 minutes to figure out the main compression algorithm, plus another 5 for the VM stuff. If you can't answer "why is 2040:1 the maximum compression ratio for the LZH backed" after looking at it for 5 minutes, well, you just suck. The PPMD backed I haven't really looked into. The actual container format is dead easy, I'm not answering your kindergarten questions.

>>16
Why is 2040:1 the maximum compression ratio for the LZH backed?

>>16
kill urself nub

>>18
The world4ch would be a much merrier place without you.

>>16
Are you counting WAHa's?

>>16
You're a bit late to this thread. And the compressor I had in the '90s did default to solid compression.

>>16
The official documentation sucks ass, and the ``opensource'' (sic) versions available from the RARlab website are in Sepples.

>>22
You call that Sepples? It only uses one or two innocuous Sepples features. You really don't need to know any Sepples to read it.

>>14
No, as in a CRC is just a hash of the data it's checking.

Still waiting on that infinite compression algorythm

>>25

main(){
return 0;
}
infinite compression.  The decompressor is left as an exercise to the reader.

>>24
It's not a hash of the password, which was your original claim, >>9-chan.

FUCKING MAGNETS, HOW DO THEY WORK?

>>28
magic

OP here. so consider another paradox: let's say you have a simple rar file. now you compress it with rar again. rar usually compresses files by 30-50%. so you compress it again and again and it's getting smaller. so the paradox: at some point it will become just a couple of bytes, how is that possible?

i think that's impossible b/c it would require infinite processing power but im not sure tho, whats your opinion guys?

>>30
WOW YEAH OMG INFINITE COMPRESSION IS POSSIBLES

>>30
That's why we have all those international conventions which prohibit using RAR any more than a few dozen times, otherwise there would be too many paradoxen.

>>30
dd if=/dev/urandom of=file.bin count=2000

Try compressing it now. I dare you!

>>30
When I was younger I knew this wouldn't work. However! I was a clever bastard, and I XOR encrypted the resulting file (with only a 32-bit key) and compressed again. Instead of wasting space (at optimal compression), I was able to get another roughly 10%. Lather, rinse, repeat--even with the same key¹ the results continued to be favourable for as long as I cared to do it, which admittedly wasn't very long, for I was doing this with a 486 and each pass was taking longer and longer.

_________________________________________________________
1. Do you know the reason for this restriction?

>>34
1. Do you know the reason for this restriction?
Because the key was of type frozenvoid*.

>>33
that's easy if your entropy pool is low, since /dev/urandom never blocks.
unless you're using a superior operating system like freebsd.

>>36
The fact that /dev/urandom never blocks doesn't mean it'll just put out a series of 0s once its entropy runs out. Random noise generated by a PRNG is just as incompressible as ``genuine'' randomness.

>>37
Random noise generated by a PRNG is just as incompressible as ``genuine'' randomness.
That is inherently untrue in principle, with a caveat. Practically speaking, though, bzip isn't going to do the trick.

The caveat being that a system with sufficiently greater resources could generate a sequence that in principle could not be compressed by a system with inferior resources, though abstractly it is still a compressible sequence.

American garbage: deflate/gzip/zlib, zip "encryption"
English mediocrity: bzip(2)
Russian glory: rar/ppmd/lzma, rar/7z encryption; Alex Ratushnyak winner of Hutter Prize and Calgary Corpus Compression Challenge

>>39
gzip + advancecomp beats bzip2 every time, and usually beats lzma.

>>40
It's difficult to tell whether you're just misinformed or just trying to stir up trouble. In any case, joke's on you: AdvanceCOMP uses the Deflate implementation from 7-Zip.

It'd have been more effective to mention KZIP, an unbeaten USA-made Deflate implementation (it's quite slow however). In any case there's no way any implementation of Deflate is ever going to beat LZMA, not even in specially crafted data. I could see how it could beat the block sorting compressors in some cases though.

It's difficult to tell whether you're just misinformed or just trying to stir up trouble. In any case, joke's on you: AdvanceCOMP uses the Deflate implementation from 7-Zip.
that doesn't change the fact that deflate usually beats your precious lzma. i don't give a shit about your country and its penile deficiencies.

In any case there's no way any implementation of Deflate is ever going to beat LZMA, not even in specially crafted data. I could see how it could beat the block sorting compressors in some cases though.
out of about 50 files (mostly c code and elf binaries) that i've tried it on, advancecomp's implementation of deflate beats lzma on all of them except one.

>>38
The only way to compress PRNG output that comes to my mind is to discover the seed used to generate the bytes - but it's not a compression then, it's a hack.

Please, explain to me, what's the difference between PRNG output and true random data? How can one tell apart them, if he does not know the algorithm used to produce it?

>>43
Speaking as someone with some NSA cryptologic experience I can assure you that it is most definitely possible to differentiate random from pseudo-random, and pseudo-random from encrypted data for that matter.

I can't delve further into this topic, but maybe someone else can.

>>44
I can't delve further into this topic
Why is that?

>>45
Because he's full of shit.

>>43
http://www.lavarnd.org/what/nist-test.html#medium

>>42
I don't think you know what are you talking about. You trolled me into testing this, I used C source files and compressed each one individually (worst case for lzma, which is designed for large amounts of data). For Deflate I used advancecomp at maximum settings. Deflate did win in some tiny files (<4KB compressed), but overall lzma was 7% smaller.

Once we use solid compression (I concatenated all the files together), lzma was leading by 28%.

I don't know why I bother - several distros and package systems have already switched to lzma, with more on the way. It's widely used on installers on other platforms too. Many embedded devices use it (my router does for example). Even friggin' WinZIP added it in a desperate attempt to remain relevant. The only reasons to use deflate are grossly underpowered hardware, or compatibility.

Next in /prog/: >>42 insists deflate is better than lzma, this time testing the common use case of compressing mp3 and jpeg files.

>>43
[i]You[i] are a hack.

http://www.iro.umontreal.ca/~simardr/testu01/tu01.html (read the paper)
http://www.phy.duke.edu/~rgb/General/dieharder.php

>>46
Ever hear of a security clearance, asshole?

I don't know how much of what I know is public-domain knowledge and how much is classified, and I'd rather not risk losing my clearance, much less life imprisonment or death.

>>48
i'm not sure if you're just trolling or what, but i'm looking at a .tar.gz and a .tar.lzma both compressed from the same tar file (one with gzip then advdef -z4, the other with lzma -9), and the .tar.lzma is about 400KB larger.

>>51
Please post that file, I'd like to take a look.