=============================================================================== SSH client notes hints and tips. =============================================================================== For general ssh usage see https://antofthy.gitlab.io/info/usage/ssh_howto.txt For SSH port forwarding, socks and networking https://antofthy.gitlab.io/info/apps/ssh_networking.txt For SSH daemon information https://antofthy.gitlab.io/info/apps/sshd_server.txt For using SSH public keys with pass-phrase protection see And Kimmo Suominen's Guide... http://kimmo.suominen.com/docs/ssh/ Or https://www.digitalocean.com/community/tutorials/ssh-essentials-working-with-ssh-servers-clients-and-keys Using SSH from windows 10 https://www.howtogeek.com/336775/ ------------------------------------------------------------------------------- SSH versions (protocols and implementation) First there are two protocols v1 and v2 The first is insecure and is NOT used anymore, anywhere. (Just as: telnet, rsh, and non-anoymous ftp; is should not be used) The other distiction is Commercial SSH and OpenSSH which are just different implementations and slight option differences, with the biggest difference being how keys are stored on disk and in homes. (see below) SunSSH v3 is basied on OpenSSH, but with some slight differences in command line options. Caution is needed for advanced scripts. ------------------------------------------------------------------------------- SSH escapes (after a newline - PTY connection only) ~? list the escapes ~^Z suspend ~. terminate ~& background (wait for forwarded connections to terminate) ~# list forwarded connections ------------------------------------------------------------------------------- SSH Public Private Key Generation and Conversion SSH Version 1 Run the command, press return for all questions ssh-keygen To allow this account to access another account without a password append the file ".ssh/identity.pub" into ".ssh/authorized_keys" on the account you which access. These keys are opsolete and only the version 2 DSA keys are now needed. OpenSSH Multiple key types can be generated for maximum interoperatibility. NOTE: do not use "rsa1" or "dsa", they are obsolete. ssh-keygen -t rsa ssh-keygen -t ecdsa ssh-keygen -t ed25519 Then for remote account access append the files ".ssh/id*.pub" into ".ssh/authorized_keys" on the remote account. Which key is used depends on the particular setup of the SSH daemon on both systems. All your personal OpenSSH stuff is saved into ".ssh" in your home. NOTE: under open ssh you can also place authorized keys into the ".ssh/authorized_keys2" file as well. I use the normal "authorized_keys" file for access from my primary accounts, while I use the second "authorized_keys2" file for local file bettwen hosts within a cluster. That second file is NOT distributed (updated) from my primary hosts. This improves security, by fine tuning which machine has access to which. NOTE: Multiple "authorized_keys" files are controled by the destination hosts "/etc/ssh/sshd_config" file setting "AuthorizedKeysFile". You can specify multiple such files. Some linux machines disable the the default use of a secondary file. Commercial SSH2 (obsolete - last used on solaris11) The commerical SSH2 keys are generated as above, but are stored in the ".ssh2" sub-directory using a different configuration, involving indirect use of individual key files. * ssh-keygen is used to generate the public and private key files EG it generates files: id_dsa_1024_a id_dsa_1024_a.pub I suggest you rename them so you know what host they are for. Eg: id_dsa_kraken id_dsa_kraken.pub * .ssh2/identification file then lists the private key files, that identify this account to other accounts. EG: it has lines like: IdKey id_dsa_kraken * .ssh2/authorization contains lines identify public key files in the ".ssh2" sub-directory which are allowed access. EG: with a line like: Key id_dsa_kobold.pub All SSH2 stuff is saved into ".ssh2" in your home. As such OpenSSH and SSH2 configuration can happilly co-exist. OpenSSH and SSH2 key conversion To convert a OpenSSH public key to SSH2 (using OpenSSH) ssh-keygen -e -f ~/.ssh/id_rsa.pub > ~/.ssh2/id_rsa_`hostname -s`.pub Now you can then add that file name to the ".ssh2/authorization" file and copy both to the machine you want to login to without a password. echo "Key id_rsa_`hostname -s`.pub" >> .ssh2/authorization To convert SSH2 public key to OpenSSH authorized keys (using OpenSSH) ssh-keygen -i -f ~/.ssh2/id_rsa_`uname -n`.pub >> ~/.ssh/authorized_keys NOTE: the base64 encoded key block does NOT care where the key is broken by newlines or spaces, so you can format it almost anyway you like, if the newlines are permitted by that file. The comment and subject fields seem to be freeform to identify which public key the file contains. Check if private and public keys match ssh-keygen -l -f id_dsa ssh-keygen -l -f id_dsa.pub ssh-keygen -l -f authorized_keys These output fingerprints for each file so you can see if they match ------------------------------------------------------------------------------- SSH Config notes. The first obtained value for each parameter is what is used... As such specific settings should be set first, and general global settings "Host *" should be set last. ------------------------------------------------------------------------------- SSH options on the command line... Due to problems with some programs (like rsync, see "rsync.txt", in this directory), most ssh programs will alow you to specify a "=" instead of a space between option and its argument. (Only openssh-2.1 does NOT seem to allow this, which caused me some problems). As such in scripts you can use... ssh -f -x -o BatchMode=yes {remote_command} Which turns off any password interaction with user (abort instead) The -f means to background the command and -x ensures an X window connection is also not setup. ------------------------------------------------------------------------------- Copying a file with a ':' in its name The following will fail as their is no host named "t" scp t:t remote: To fix you must have a '/' somewhere before the colon, in which case ssh will consider it a filename. EG: use any of the following... scp ./t:t remote: scp /path/to/file/t:t remote: ------------------------------------------------------------------------------- Pipe in a password This is not allowed... echo 'password' | ssh user@somhost command Use "sshpass" sshpass -p password ssh user@somehost command That is insecure, but you can use -f to read from say stdin echo 'password' | sshpass -f /dev/stdin ssh user@somehost command Or use SSHPASS environment variable. putty-tools (plink) plink user@domain -pw mypass [cmd] passh Its a sort of 'expect' type progrm. https://github.com/clarkwang/passh DIY shell... You can use SSH_ASKPASS environment variable to pass a passwor to ssh, if ssh has no tty! You can force a no tty, using setsid... But you need to handle the prompt argument to the askpass.. =======8<--------CUT HERE---------- #!/bin/bash # # echo 'passwd' | pass ssh user@host command # if [[ $1 =~ password: ]]; then cat else SSH_ASKPASS="$0" \ exec setsid "$@" # disconnect from tty fi =======8<--------CUT HERE---------- Using Perl to call SSH... =======8<--------CUT HERE---------- use Net::SSH::Perl; my $ssh = Net::SSH::Perl->new('$host'); $ssh->login('$user', '$pass') or die "Oh noes! $!"; =======8<--------CUT HERE---------- Using expect to handle password... =======8<--------CUT HERE---------- #!/usr/bin/expect -f # # ./ssh_expect password 192.168.1.11 username # set pass [lrange $argv 0 0] set server [lrange $argv 1 1] set name [lrange $argv 2 2] spawn ssh $name@$server match_max 100000 expect "*?assword:*" send -- "$pass\r" send -- "\r" interact =======8<--------CUT HERE---------- ------------------------------------------------------------------------------- SSH Host Aliases This is especially useful to simplify complex connections or for host with long names or domains. =======8<-------- # Define how to ssh to 'home' Host home User anthony Hostname my-home-dynamic-dns-name.hopto.org Port 222 HostKeyAlias laptop.example.org =======8<-------- Now I can just use ssh home instead of ssh anthony@my-home-dynamic-dns-name.hopto.org:222 This becomes even more useful when creating a jumpbox to get into or out of a secure intranet. ------------------------------------------------------------------------------- SSH to a Dynamic (roaming) IP Address. If the machine you are trying to ssh to changes IP address, the following options allow you to select the right host key, without relying on the IP address. ssh -o CheckIP=no -o StrictHostKeyChecking=no -o HostKeyAlias={machine_name} {Current_IP_of_machine} The host key to identify that machine can then be set correctly. This is useful if you want to SSH back to a machine, which dialed onto the network via a PPP modem or dynamic ADSL connection. That is ssh to where you logged in using information found in in "$SSH_CLIENT" or "$SSH_CONNECTION" environment variable. ------------------------------------------------------------------------------- Persistent SSH Multiplexed Connection Host bastion bastion.example.com # SSH hostname for alias 'bastion' User s1234567 Hostname bastion.example.com # TTY is needed, X is not RequestTTY yes ForwardX11 no # Reuse an existing connection (faster, no need for password again) # Persistent until it has had no connections for more than a hour ControlMaster auto ControlPersist 1h ControlPath ~/.ssh/sshctl_bastion #ControlPath ~/.ssh/sshctl_%r@%h:%p #ControlPath /tmp/ssh_%u/%r@%h:%p # if directory has been created # Once you have authenticated into the 'bastion' server once, you will re-use that connection, and not need to authenticate again. Connection Controls... ssh bastion -O check # Check if multiplex connection is still running ssh bastion -O exit # Stop, Disconnect all multiplexed connections ssh bastion -O stop # No new connections, close when last connection exits ------------------------------------------------------------------------------- Double ssh jump ssh s1234567@gateway.example.com ssh user@internal.example.com This does a FULL ssh to the gateway host, and then goes from there to the internal host. That is it is the gateway account that authenticates to internal account, NOT the starting account. See Cavat on using SSH Proxy options below, as to why you would use this. Double ssh jumps can be important to bastion servers which sets the authority to access accounts on internal hosts for individual users. For example special signed SSH principles. See "https://antofthy.gitlab.io/info/apps/ssh_principles.txt" Note: if you are using a wrapper you will have add extra quoting to the arguments being passed.. # assuming remote command is set in "$@" ssh s1234567@gateway.example.com \ ssh user@internal.example.com \ "${@@Q}" ------------------------------------------------------------------------------- Proxy via 'jumpbox' or 'firewall' host. Using options ProxyHost and ProxyCommand, as well as -J option. This launches a command on a 'jumbox' host that creates a network port connection to the remote server. However a Proxy Jump authenticated using keys from the source host, for BOTH the jumphost, and the final internal host. Using Proxy Jump option... ssh -v -o ProxyJump=s1234567@gateway.example.com \ user@internal.example.com Newer SSH has a CLI option, -J for ProxyJump ... sh -v -J s1234567@gateway.example.com \ user@internal.example.com Using ProxyCommand and ssh -W via CLI ssh -o ProxyCommand='ssh -qax -W %h:%p s1234567@gateway.example.com' \ user@internal.example.com Using ProxyCommand and netcat via CLI ssh -o ProxyCommand='ssh -qax s1234567@gateway.example.com nc %h %p' \ user@internal.example.com Cavats.... * Both the gateway and internal account is authenticated using the information supplied by the starting account where ssh was initiated. It does NOT use ssh keys, or ssh configuration on the gateway host! * The ProxyJump will not use account or keys that may be present on the gateway. * ProxyJump must be able to DNS resolve the internal hostanme on the starting host. * ProxyCommands using 'ssh" do not pass on verbose flags, unless given as part of an ssh proxy command. --- Setup in ssh config... =======8<-------- # Proxy to internal hosts via a 'jumpbox' Host *.example.com !gateway.example.com # This is best as it will also forward any verbose flags for debugging # However I have found MOTD's are often still printed. # This is equivelent toi using the '-J' option on the commands line ProxyJump jumpbox # The equivelent command to ProxyJump (other than flag passing) #ProxyCommand ssh -qax -W %h:%p -o TCPKeepAlive=yes jumpbox # Verbose Version for debugging #ProxyCommand ssh -vax -W %h:%p -o TCPKeepAlive=yes jumpbox # Using netcat to setup the proxy instead of ssh -W #ProxyCommand ssh -qax jumpbox nc %h %p #ProxyCommand ssh -qax jumpbox ncat %h %p #ProxyCommand ssh -qax jumpbox socat STDIN TCP:%h:%p # Define the jumpbox server, destination username is preserved Host jumpbox User s1234567 HostName gateway.example.com =======8<-------- An alternative using 'Match' to only proxy if the host in that domain is not directly reachable. Assuming the jumpbox itself is reachable! =======8<-------- Match Host *.example.com !exec "nc -z -w 1 %h %p" ProxyJump jumpbox =======8<-------- You can then ssh direct, and the config does all the work ssh user@internal.example.com ------------------------------------------------------------------------------- Chained ProxyJumped Host Paths Using '/' EG: jumphost/bastion/server =======8<-------- Host */* ProxyCommand ssh -W $(basename %h):%p %r@$(dirname %h) =======8<-------- Using '^' and bash variable manipulation =======8<-------- Host *^* # Either ... #ProxyCommand ssh -qax -W $(h=%h; echo ${h%%%%^*}) $(h=%h; echo ${h#*^}) # OR... ProxyCommand ssh -qax $(h=%h; echo ${h%%%%^*}) nc $(h=%h; echo ${h#*^}) %p =======8<-------- Using '+' and sed to separate parts with ports EG: host1+host2:222+host3:2222 NOTE however that any ':' WILL confuse things like sftp, scp and rsync =======8<-------- Host *+* ProxyCommand ssh -W $(echo %h | sed 's/^.*+//;s/^\([^:]*$\)/\1:22/') $(echo %h | sed 's/+[^+]*$//;s/\([^+%%]*\)%%\([^+]*\)$/\2 -l \1/;s/:\([^:+]*\)$/ -p \1/') =======8<-------- ------------------------------------------------------------------------------- DNS is intranet accessible only If the destination hostname cannot be DNS'ed outside the intranet then the ProxyCommand may be needed, to 'hide' the destination hostname =======8<-------- ssh -o ProxyCommand="ssh -W server.example.org:22 jumphost.example.org" \ user@server =======8<-------- In this case on the local machine 'server' is only used to lookup the appropriate hostkey/public keys needed. Only on the 'jumphost' is 'server' looked up in the DNS. ------------------------------------------------------------------------------- Host Key Rotation Client /etc/ssh/ssh_config and ~/.ssh/config UpdateHostkeys=ask This gets ssh to ask for new (or alternative) hostkey updates. The protocal will verify that the client actually owns the private key. Old keys no longer avaiable are removed from known_hosts Server sshd_config # Old keys (to be expired) HostKey /etc/ssh/ssh_host_rsa_key HostKey /etc/ssh/ssh_host_dsa_key # New keys HostKey /etc/ssh/ssh_host_rsa_key.new HostKey /etc/ssh/ssh_host_ed25519_key HostKey /etc/ssh/ssh_host_ecdsa_key Once enough clients have picked up the new hostkeys, the old ones can be removed, and the clients will remove them from known hosts files. Source http://blog.djm.net.au/2015/02/key-rotation-in-openssh-68.html ------------------------------------------------------------------------------- SSH Path and Environment Check what path is used on the remote server and that there is no other output that could interfer with the connection.. ssh remote.domain 'echo $PATH' or the rest of the envionment ssh remote.domain env You can override the initial ssh path my adding it to ".ssh/environment" on the remote server, but this only works if "PermitUserEnvironment" is set to true in the "/etc/ssh/sshd_config" file. ("No" by default) echo "PATH=$PATH" >>~/.ssh/environment If this is not-posible (typical) you can try something like... ssh remote.domain /bin/sh -c 'export PATH=$PATH_TO_USE; cmd' And debug (using pstree) with... ssh localhost 'pstree -p $$' ssh localhost '/bin/sh -c '\''pstree -p $$'\' ... and so on ... Note that a better way of setting the environment is to use the shell and not the ssh, or the remote command. T/Csh's will read the ".cshrc" file which can abort early BASH will read ".bashrc" file (only uses it for non-interactive logins) In both cases the configuration can abort after non-interactive shell setups have been completed. (PS1 is not defined if bash is non-interative! That is you set PATH and and required environment, without doing any of the other shell setups like prompts, aliases, functions, etc. ------------------------------------------------------------------------------- SSH and the login shell When running any command ssh requires a valid login shell. One that allows a "-c ..." option to execute the command. This includes... * command passed in the ssh command line (i.e. ssh user@host command) * command set up in the .authorized_keys file (i.e. command="..." ssh-rsa AAAA...) * command specified by the /etc/ssh/sshd_config" "ForceCommand option. * executing the "~/.ssh/rc" script (using... "shell ~/.ssh/rc" ) * The scp and sftp-server commands for SSH provided file transfer (the path of sftp-server is set as a subsystem in "/etc/ssh/sshd_config") For more information see... https://antofthy.gitlab.io/info/apps/ssh_remote_commands.txt WARNING: You do NOT need a valid local shell to set up a remote port-forwarding link! Unless the remote PAM configuration includes a "pam_shells" check. ------------------------------------------------------------------------------- Do not output a banner Client Side. ssh -o LogLevel=Error ... Server side touch ~/.hushlogin SSH Server Config vi /etc/ssh/sshd_config Banner /dev/null # or PrintMotd no systemctl restart sshd ------------------------------------------------------------------------------- Using a SSH Hold Command This was to keep things like forward ports, or proxies open, and prevent the backgrounded ssh command from exiting. This is no longer needed, as you can now use a "-N" flag to specify that no command shell is needed on the remote host! With a socks proxy, or port forwarding I usally like to have the ssh command run a 'holding program on the remote server. This command holds the port, to ensure it does not unexpectantally close. One command I have used (for a X windows connection) is... exec xlogo -name proxy -g 20x20-5+5 -fg red -bg navy \ -xrm 'proxy*baseTranslations: #override : quit()' I also sometimes use the color reversal, to denote the reverse direction... exec xlogo -name proxy -g 20x20-30+5 -fg blue -bg firebrick \ -xrm 'proxy*baseTranslations: #override : quit()' The 'xlogo' program is an X windows command that should exist on any machine with X windows capability. This means the command will automatically exit and die, closing the remote connections, when you log out of your machine. The special 'resource option' or -xrm is designed to let you close the connection by just pressing the mouse button on that small window. ------------------------------------------------------------------------------- Keeping ssh keys on USB key only. Limiting access to a server to ssh pubic keys only is in many ways far more secure than using passwords. Some secure servers only allow ssh-key login and disable user password logins completely for improved security. This was from an artical on how you can store local password protected keys on a USB key to provide remote access to a server and via that a separate root ssh key access. NOTE: It talks about using this as a two factor authentication (password for key and the key itself). However the server itself only sees the key, not the password so it is actually only really a single factor security. From Linux Journal Tighter SSH Security with Two-Factor Authentication http://www.linuxjournal.com/article/8957 By only allowing SSH public key logins which must be stored en encrypted form, will provide increased security without the use of static passwords on the remote machine. User authentication... Prepare a USB stick that holds your encrypted SSH public keys, each public key in a separate file that us named appropriatally for access to a specific machine. You can then login to the remote host using the -i option to specify your encrypted identity file. ssh -i /media/usbdisk/key-rsa-bob@machine2 bob@machine2 Only you with the decrypting password for that key can use them, and they are only stored on a USB stick. ASIDE: this is not really two factor as one is 'local' not on the server. User can also decrypt and copy the key to his account freely. Root Authentication... For root, you do not want to allow remote root access, so first set the ssh deamon to also look at the localhost loopback interface then only allow root via the loopback interface only. To make this work you would use ssh-add to add the key for root to the ssh-agent (typically already running), and then you can ssh to root on the remote machine you have already logged in on. EG in sshd_config you have something like PermitRootLogin yes AllowUsers bob@* AllowUsers root@127.0.0.1 and restart. Example get root on remote machine... First the user on his current machine does setup root access (5 min limited time access) ssh-add -t 300 /media/usbdisk/key-rsa-root@machine2 then login (as above) ssh -A -i /media/usbdisk/key-rsa-bob@machine2 Then on the remote machine... See that we have access to the roots key ssh-add -l and become root (or run a command as root) ssh root@localhost Note that the roots private key never leaves the users USB key or the machine he is currently using. No one can get root on the remote machine without that USB setup. Now that you can two-factor auth login to root you can lock down its normal password access, and even lockdown su and sudo access The ssh-agent on users machine however should also be locked down using -c so that the user on the original machine must confirm the remote machines use. This is protection against a lurker that has already cracked the remote machne and he wants to hijack root keys to other machines. An alternative is a seperate ssh-agent for each machine and set the enviroment variable SSH_AUTH_SOCK appropriatally before connecting to the remote machine. Of course that should get wrappered into a script "tfssh" tfssh [username@]host [keydir] Download from ftp://ftp.linuxjournal.com/pub/lj/listings/issue152/8957.tgz Saved in ~/store/scripts/net_connect/two_factor_ssh ------------------------------------------------------------------------------- SSH X windows with no home When you have no writable home, sshd can not create the .Xauthority file for X Windows. BUT you can get it to create it elsewhere using a "/etc/ssh/sshrc" script... =======8<-------- #!/bin/sh # # Set up a Xauthority for users that DO NOT have the HOME set correctly # # NOTE: this can NOT set the XAUTHORITY environment variable # so it points to the created "/var/tmp/Xauth_$USER" file # # Anthony # #if ...home directory not writeable...; then # if read proto cookie; then # echo "add $DISPLAY $proto $cookie" | # /bin/xauth -q -f /var/tmp/Xauth_$USER - # fi #fi =======8<-------- The "/etc/profile" will then need to set the environment variable =======8<-------- export XAUTHORITY=/var/tmp/Xauth_$USER =======8<-------- ALTERNATIVE... Create a wrapper around "xauth" =======8<-------- #!/bin/sh # # Override the location of the Xauthority file. # That way when "sshd" or others call it it locates # the authority in a common (more secure) location. # XAUTH=/bin/xauth-orig # location of real xauth command PATH=/bin:/usr/bin HOME=/var/tmp/.Xauthority-"`whoami`" XAUTHORITY=$HOME/Xauthority export HOME PATH XAUTHORITY ( umask 077 && mkdir -p $HOME && chown "`whoami`" $HOME ) && $XAUTH "$@" chmod 700 "$HOME" chmod 600 "$XAUTHORITY" =======8<-------- You still need to set the XAUTHORITY for the user in "/etc/profile" Techniques found on https://answers.ssh.com/questions/523/ -------------------------------------------------------------------------------