------------------------------------------------------------------------------- Hashing: creating a irreversible 'hash' or 'checksum' of some text ------------------------------------------------------------------------------- Hashing, what use is it... True 'hashing' is a one way encryption that is used for many tasks... Password Storage (Password Files)... It is not a good idea to store a users password such that it is recoverable. As such a hash of the password is stored. The hash should prevent anyone 're-creating' the users original password from the stored hash. To comparing passwords, the users just input password is also hashed in the same way, and the hashes compared to see if the password match. Expand User Passwords into Encryption Keys... This form of 'hashing' is also often used to take short user password, and expand them into a longer 'cryptographic key' that is needed for actual encryption work. However it would still be a 'weak' password, as such it should really only be used to decrypt a strong 'master' key (randomally generated), that is used for actually encrypting large amounts of data. See "Password encrypted master passwords" below. Checksumming (anti-tampering)... That is the generation of a smaller string that represents a large data block, such as a message or file. If that data is modified or added to in any way the hash of the data will be different. The hash can then be regarded as a 'fingerprint' or verification that the data was not changed in some specific way. Cryptographic hashes are designed to make it very hard to generate a specific hash from 'designed' input data. Also it should prevent you from changing the data in a specific way, then adding extra data to the end so as to generate the same original hash. Identifying... That is the checksum of the data is then used to 'locate' or 'lookup' the desired data from from database. However it should be noted that hash'ing is not a compression. You can not store 5000 bytes of data in a 64 byte 'hash'. Because of this hash'ed checksums can not gurantee that two completely separate data strings does not generate the same hash. It does however make the chance of a 'clash' happening quite small, especially if two hashes, or some other 'source' identification is also included. Examples... * Determining if a one file is a copy of another file. For example as a dirty means of file comparing lots of files. Hashing can be used to identify posible matching files, but file data should then be compared byte-by-byte before declaring any two files really do match. * Freenet uses hashes to identify files in its network (cloud) distributed filesystem. All files are referenced via URL-like references of those hashes to index to locate and download that file to other users. It mitigates 'hash clashing' by using two separate hashes for the identification. ------------------------------------------------------------------------------- Passwords into Hashed Passwords (Crypt)... Or use 'crypt()' -- man crypt Use perl crypt() with a salt of '$1$........$' where . are random characters of 0 to 16 characters For example. perl -e 'print crypt( "sys", "\$1\$passwd\$" ), "\n"' $1$passwd$1SBMpWvDStmWt1pAKy9D/1 Strip the 'salt' from the string (to thrid '$') PHP can do too php -r 'echo md5("test");' # MD5 hashing Openssl can also do this openssl passwd # Des encryption (-salt for compare) openssl passwd -1 # MD5 hashing function Or an openssl 'nosalt' encryption can be used as a type of hashing function The input data is just a fixed 'salt' (the -a option generates base64 output) echo "data_to_encrypt" | openssl aes-256-cbc -a -pass pass:passwd -nosalt The password could be used to encrypt itself as a type of nosalt hash but that is not recomended. ------------------------------------------------------------------------------- Reversible Password Hash Simple Encryption methods for passwords in scripts and programs. This is not really 'hashing' just basic information hiding. These are methods of encrypting a small text passwords so its 'string' is not reconisable as being a password in a compiled program. uuencode base64 # see ~/info/misc/mime_base64.hints openssl enc # full text encryption (password to protect password!) # basically swap data and password in last example above ------------------------------------------------------------------------------- Password encrypted master passwords See "Encrypted Key on Device itself" in ~/info/crypo/linux_dmcrypt.hints or the original paper by Clemens Fruhwirth, TKS1, An anti-forensic, two level, and iterated key setup scheme. http://clemens.endorphin.org/TKS1-draft.pdf That is you have a table of binary master passwords which you decrypt/hash in some way to produce the original password. This relies on the original system being encrypted with the random binary password which is then separtally encrypted using a users password in some form of 'key store'. This then allows you to have multiple completely separate passwords decrypting the same master key to the same set of data. The above scheme was incorperated into the "LUKS" version of dm-crypt, and uses a table of 8 slots and with the user provided password 'tried' against using ALL the slots, to see if it can decode the binary key needed to unlock the data. A different system is use a hashing table, where a 'data identifier' is hashed to find a encrypted 'key file'. This 'key file' is decrypted using the users password. That 'key file' can contain not only the long binary master password, but all the other details of the encryption, such as method, data location, and any other configuration information needed (salt, iteration count, config files, etc). This allows the same name/password pair to use ANY encryption method to decode any set of encrypted data. The location of the 'key files' (or 'key store') can be separated from the actual encrypted data, even physically, or via network! And different 'key stores' could decrypt different sets of data using the same name/password pairs, or even destroy that data! I have implemented this is a "ks" (or key_store) script. http://www.imagemagick.org/Usage/software/#ks That is only the start of the KS system however, but it is essentually equivelent to a LUKS password to master key system. ------------------------------------------------------------------------------- PBKDF2 (RFC 2898) algorithm Password-Based Key Derivation Function v2 Slows down password to Key Derivation (Hashing) so as to not be a burden for legitimate parties, but generate a significant burden for opponents. Basically generating iterative hashes of hashes in such a way that it takes about 1 second to compute the final hash on a modern computer. This generally used for large file decryption, not for one-time pad communications. This requires: password, iterative_count, salt, and the hashing function that is to be used (usally builtin). Only the first of which needs to be kept secret. Of course the actual encryption also requires the encryption scheme used, and a salt and iv, which also does not need to be kept secret. The OpenSSL library provides such a function called PKCS5_PBKDF2_HMAC_SHA1(). However it is not currently directly available outside its C API. A trivial C example program "pkcs5.c" was found on the OpenSS forum, whcih I modifed to generate a "pbkdf2.c" to break out this function from the openssl library to command line (and thus both shell and perl access). See http://www.cit.griffith.edu.au/~anthony/software/pbkdf2.c A Perl Crypt::PBKDF2 module was also found implementing the PBKDF2 algorithm completely in perl. However it has far too many 'non-standard' dependancies producing a 'dependancy hell' if I wanted to use it. However another PBKDF2 function also written purely in perl (and proved to be equivelent to the OpenSSL) was found in the Palm::Keyring module on CPAN See http://www.perlmonks.org/?node_id=631963 This has been converted to a standalone perl script (or include module) See http://www.cit.griffith.edu.au/~anthony/software/pbkdf2.pl This function has also been incorperated into my "encrypt" and "ks" perl script. It is slower than using the command line "pbkdf2" program (see above), but removes the external dependancy. I will now use it until such time as PBKDF2 become properly available in perl. ------------------------------------------------------------------------------- OpenLDAP Password Hash.... From Exporting MD5 from LDAP to Shadow http://lists.fedoraproject.org/pipermail/389-users/2009-January/008805.html A password in OpenLDAP slappasswd -h {MD5} -s "heslo" {MD5}lV2wuB7xmJtKTf6ugGGppg== Ignoring the pad, the two passwords are the same length (22 chars) A direct md5 hash (as hex) is echo -n "heslo"|md5sum 955db0b81ef1989b4a4dfeae8061a9a6 echo -n "heslo"|openssl dgst -md5 -hex 955db0b81ef1989b4a4dfeae8061a9a6 The OpenLDAP passwd converted to hex is... echo '' | php 955db0b81ef1989b4a4dfeae8061a9a6 OR direct convert the md5 hash as base64 (via PHP) we get... echo '' | php lV2wuB7xmJtKTf6ugGGppg== Which is the same as the OpenLDAP password. That is OpenLDAP password is simply a direct md5 password in base64 -------------------------------------------------------------------------------