Encrypting ---------------- This is a FYI and you probably should keep this for future use. Any sensitive information, such as passwords both system and private should be stored in encrypted files, and NOT as plain text. This guide is for encrypting individual files. Other encrypted data techniques beyond single individual files can also be used. For example... encrypted block devices (whole disks, partitions, disk files) encrypted directories, such as EncFS (store on any device, including cloud) Tha latter saves the contents of directory trees in multipel encrypted files. As it is file based it can be used for mounted file systems, such as university file systems, and cloud storage, such as dropbox. This system is what I use for most of my encryption needs. For more advanced examples with specific methods see the file on those methods... https://antofthy.gitlab.io/info/crypto/openssl.txt https://antofthy.gitlab.io/info/crypto/perl_encrypt.txt https://antofthy.gitlab.io/info/crypto/public_keys_gpg.txt https://antofthy.gitlab.io/info/crypto/public_keys_openssl.txt https://antofthy.gitlab.io/info/crypto/vim_encryption.txt ------------------------------------------------------------------------------- Encrypted File notes... Things that could (or should) be included with the raw encrypted data, in a encrypted file... 1/ Some 'file magic' to identify the encryption method. Though strictly not required, and tells a cracker what encryption scheme has been used, it is required if you want to implement multiple versions or variations of file encryption. That is for backward compatibility. OpenSSL file encrupting learned this the the hard way, when they improved their encryption with Salting. It now uses "Salted__" as its encryptions "file magic" However OpenSSL does not store the actual encryption scheme or MAC method used for password to key, or data varification (whcih is doesn't do). As such its "file magic" is not really complete. 2/ Random salt (public) for the password hashing. This makes a static 'rainbow table' hashed dictionary words useless. 3/ PBKDF v2 iteration count (public) for password to encryption key. That is the public count of how many times the hashing function should be applied to the user supplied key to convert it into a cryptographic key (the actual binary key used in encryption). This interative hashing vastly improves security. First by adding more complexity to the final cryptographic key, and by my making it a lot slower to crack the password without making it two slow for a user to really notice. That is, it adds about 1/2 to 2 seconds delay, to hash the users password, whcih for users is acceptable, but makes dictionary attacks unfeasible. 4/ A verification identifier (encrypted or hashed). Essentually something that will verify whether the user's passphrase is actually correct, and will decrypt the file data correctly. This is NOT to provide a checksum of the whole file, though that can also be used to verifiy the password as well as the data. It's purpose is to verify users password in a timely manner, before doing all the work to decrypt a posibly extremely large amount of encrypted data, as well as prevent the decrypt command spewing binary garbage with unknown consequences. This verification identifier could be... (a) a 'fixed string' (without salting this could weaken encryption) (b) the users original (unhashed) password (c) the cryptographic hash used for decryption (d) the salt (its random, so should be a good test) (e) Some type of file checksum. For example, the checksum of just the first block of the decrypted data, so it can be checked early. Note that any verification indentifier can make the file encryption easier to crack. But it generally does not help the cracker very much, as most dictionary attacks (and humans) look for 'intelligible results' to determine if decryption is successful. It should not be used if some form of double encryption is used, though double encryption is generally considered wasted effort. 5/ MAC - Message Authentication Code Is the encrypted data correct and unaltered. -- Authenticated Encryption Often this final data integrity verification is easier to do by the program that is actually making use of the data, rather than as part of the encryption process. Warning: This can be difficult to implement in a data pipeline as the data will have an unknown length. Also in any type of pipeline technique, any such MAC will always need to be added at the end, rather than in the header, and then you have the problem of determing end of data, relative to the checksum. Security Warning: Do not use a MAC of plaintext. http://www.thoughtcrime.org/blog/the-cryptographic-doom-principle/ Idealy MAC the encrypted text and append that, so you can check the message before attempting to decrypting (and failing) of course that means you need the whole message before hand, which does not work for pipelined data. Specific examples... OpenSSL The original "OpenSSL enc" used none of the above when it was first released. It later added a hashing salt (2), as well as some file magic (1), so it can automatically identify the new encryption format, without breaking the older existing openssl encrypted files. The "openssl enc" v1.1.1, "-pdkdf" option while adding (3) does not store the interation count ("-iter") in the file header, so whether it is actually used, and the actal count, (which should change as computers get faster), has to be stored externally. :-( This change also made storing the metadata required for decrypting a problem, as previous defaults, were no longer defaults! See "keepout" wrapper below. A perl "encrypt" program https://antofthy.gitlab.io/software/#encrypt Is very similar to the "openssl enc" but performs PBKDF2 (3) when converting the passwd to a cryptographic key and IV. However it does not provide any form of 'verification' (4 & 5), at least not yet. However it also did not store the cypher or the digest used for hashing. As such it has become dated. The "keepout" shell wrapper for "openssl enc" https://antofthy.gitlab.io/software/#keepout This is a wrapper around openssl (preferably v1.1.1 and above) that adds an extra header (much like the old popular "bz3aespipe" did) holding all the options used by "openssl enc" command to encrypt, and thus needed to decrypt the data. It is a simple plain text header, that can be update and expanded as "openssl" changes (and it is likely to) in the future. The "dmcrypt", or "LUKS cryptsetup" used for disk partition encryption under linux, implements 1 to thru to 4. The fifth is automatically provided by the filesystem stored on the encrypted disk, so all 5 aspects is handled. Similarly normal use of "EncFS" also provides 1 thur to 4 using information in the configuration file, that is normally stored in the parent directory of the encrypted directory tree. But the 4th item may be ignored if a 'any password' option is enabled. Storing the configuration file with the encrypted volume is a major weakness in that anyone with access (eg: in the cloud) could modify the information so future saves are saved in a less secure fashion. ------------------------------------------------------------------------------- VIM Editing of Encrypted files See https://antofthy.gitlab.io/info/crypto/vim_encryption.txt =============================================================================== Specific methods of file encryption... ------------------------------------------------------------------------------- OpenSSL encrypted files (basic example) See https://antofthy.gitlab.io/info/crypto/openssl.txt for better more up-to-date examples. Salted AES encryption (this is very common) openssl enc [-d] -aes-256-cbc < file.txt > file.aes WARNING: This uses the fast PBKDF1.5 password to key function. Which is a good hashing function for fast generation of one time pads. But it is not good for long term encryption. such encryption should use the PBKDF2 password hasshing method (essentually an interative hash) that slows up decoding and thus dictionary attacks on passwords. See next... Note output is salted (randomised). To get a deterministic (unchanging) output you will need to use "-nosalt", which removes both the file magic and the random salt before the actual encrypted data. As of OpenSSL v1.1.1 you get warnings to use "-pbkdf2" and "-iter {count}" (with "-md hash") to implement the use of better (slower) password hashing, for generating the cryptographic password. The "-iter" count especially (default is still very low), may be highly variable. openssl enc -md sha512 -pbkdf2 -iter 500000 -aes-256-cbc \ < file.txt > file.openssl Decryption is exactly the same but with '-d' flag added. Basically extra information will need to be either programmicaly configured, or saved as meta-data with the encrypted file. Including... * If "-pbkdf2" was used * The "-iter {count}", the default is very low, and should be increased * the "-md {digest}" used for password hashing (it changed in v1.1.0!) * The encryption scheme The only thing OpenSSL saves in the encrypted file is... * A 8 character file magic of 'Salted__' * A randomised "Salt" (4 bytes, or 32 bits) for password hashing * The encrypted data ALL other options used during the encryption are 'assumed' to be known, which means the information needs to be either known, or saved in some way. See the "keepout" script for one such method. https://antofthy.gitlab.io/software/#keepout --- Using "openssl" via a program. Openssl has a very versitile "-pass" option to define where to get the users password from. Few other programs have this same versitility for password input. It can read passwords from... * command line, "-pass pass:PASSWORD" -- unsafe * an environment variable "-pass env:VAR" -- unsafe * the first line from a file or pipe * a file handle "-pass fd:N" or "-stdin" (given before the data is read) The last is the most secure method but tricky to use... For example using BASH 'here file'... openssl enc -salt -aes-256-cbc -md sha512 -pbkdf2 -iter "$iter" \ -pass fd:4 4<<<"password" to decrypt use the exact same command with an added "-d" option The openssl wrapper script "keepout" can do this... https://antofthy.gitlab.io/software/#keepout But also provides the means to use keyring caching of passwords for use during opertions such as 'file editing' or other 'user sessions'... https://antofthy.gitlab.io/info/crypto/passwd_caching.txt https://antofthy.gitlab.io/info/crypto/keyring_linux_kernal.txt ------------------------------------------------------------------------------- Mcrypt file encryption (DO NOT USE -- abandonware with bugs and old practices) Mcrypt offers a lot more controls over the generated file format. The most important options can be pre-set using command line, environment variables, or a configuration file (rc config, or specified). Note the file checksum method can not be set that way. The 'key' may be specified in a number of ways. such as: command line option, environment, keyfiles. Though not pre-opened file descriptors. It can even be given as a raw pre-encrypted hex key. Essentually it is a more controllable form of the openssl file encrypt. mcrypt --keymode s2k-salted-md5 \ --algorithm rijndael-256 --mode cbc file or MCRYPT_ALGO=rijndael-256 \ MCRYPT_MODE=cbc \ MCRYPT_KEY_MODE=s2k-salted-md5 \ mcrypt file NOTE rijndael is the older original name for AES A lot of information is stored in the header of the file for easier decrypting, however a '--bare' form will remove all header information, and checksum. In that case this flag and all encryption info needs to be provided for decryption. However it does not provide access to a PBKDF2 iterative hashing function, to slow down brute force attacks. ------------------------------------------------------------------------------- PGP/GPG file encryption PGP (Pretty Good Privacy) and GPG (Gnu PGP) is normally used for sending and recieving encrypted mail, but it can also be used for encrypting and decrypting files with individual passwords. The following are details on how you can encrypt and decrypt files using the PGP system. Symmetric Encryption... First you do NOT need to have a PGP public/private key for encrypting files, such keys are generally for the sending of encrypted mail, not files. However GPG will insist in creating empty keyrings, even though they are not needed. pgpe -fc < file.txt > file.pgp # encrypt PGP pgpv -f < file.pgp > file.txt # decrypt PGP gzip < file.txt | pgpe -fc > file.pgp # encrypt PGP GZipped pgpv -f < file.pgp | gzip -d > file.txt # decrypt PGP GZipped gpg -c -o - < file.txt > file.gpg # encrypt GPG AES-128 gpg -c -o - --cipher-algo=AES256 < file.txt > file.gpg # AES-256 gpg -o - < file.gpg > file.txt # decrypt GPG gzip < file.txt | gpg -c --force-mdc -o - > file.gpg # encrypt GPG GZipped gpg -o - < file.gpg | gzip -d > file.txt # decrypt GPG GZipped Files encrypted with PGP are saved with the suffix ".pgp" (encrypted with IDEA) while GPG files are saved with ".gpg" (typically CAST5 encryption). Such a file is generally a binary file, though a option exists to create a ascii version (known as ``ascii armoured'') which could then be safly mailed. To do this a "-a" option is also given during encryption, no change to the decryption is needed. One advantage of using gpg is that it will remember a password used to decrypt a gpg file for a period of time. The password is held by "gpg-agent" which is automatically started as needed. This means you can use the contents of a gpg file multiple times and only need to enter the password once... EG: This only needs one password entry... ASIDE: Contents should not end in a newline for cryptsetup! gpg -qd luks_passwd.gpg | sudo cryptsetup luksOpen /dev/sdx1 sdx1 -d - gpg -qd luks_passwd.gpg | sudo cryptsetup luksOpen /dev/sdy1 sdy1 -d - gpg -qd luks_passwd.gpg | sudo cryptsetup luksOpen /dev/sdz1 sdz1 -d - Kill the gpg-agent to ensure password is forgotten gpgconf --kill gpg-agent The "gpg-agent" uses "pinentry" as a password helper. You can set exactly which program should be used using gpg config option "pinentry-program". See also the vim code to let you directly edit encrypted files... https://antofthy.gitlab.io/software/#encrypt.vim And https://antofthy.gitlab.io/info/crypto/vim_encryption.txt Public Key Encryption... PGP or GPG also allows you to encrypt/decrypt a files using pre-prepared public/private keys. You do not need passwords to encrypt the file, as the public key is used. But you will need a password to decrypt the file. For command information see. https://antofthy.gitlab.io/info/crypto/public_keys_gpg.txt Caution... The speed of failure for a wrong password on a private key makes me doubt that GPG is using hash iteration (PBKDF2) to protect the pass-phrase to cryptographic, key stretching. Just how secure are private keys and symmetric encrypted files? -------------------------------------------------------------------------------