Automate scp file transfer using a shell script

I have some n number of files in a directory on my unix system. Is there a way to write a shellscript that will transfer all those files via scp to a specified remote system. I'll specify the password within the script, so that I don't have to enter it for each file.

Instead of hardcoding password in a shell script, use SSH keys, its easier and secure.

$ scp -i ~/.ssh/id_rsa *.derp devops@myserver.org:/path/to/target/directory/

assuming your private key is at ~/.ssh/id_rsa and the files you want to send can be filtered with *.derp

To generate a public / private key pair :

$ ssh-keygen -t rsa

The above will generate 2 files, ~/.ssh/id_rsa (private key) and ~/.ssh/id_rsa.pub (public key)

To setup the SSH keys for usage (one time task) : Copy the contents of ~/.ssh/id_rsa.pub and paste in a new line of ~devops/.ssh/authorized_keys in myserver.org server. If ~devops/.ssh/authorized_keys doesn't exist, feel free to create it.

A lucid how-to guide is available here.

#!/usr/bin/expect -f

# connect via scp
spawn scp "[email protected]:/home/santhosh/file.dmp" /u01/dumps/file.dmp
#######################
expect {
  -re ".*es.*o.*" {
    exp_send "yes\r"
    exp_continue
  }
  -re ".*sword.*" {
    exp_send "PASSWORD\r"
  }
}
interact

http://blogs.oracle.com/SanthoshK/entry/automate_linux_scp_command

why don't you try this?

password="your password"
username="username"
Ip="<IP>"
sshpass -p "$password" scp /<PATH>/final.txt $username@$Ip:/root/<PATH>

you could also use rsync. It seems to work better for multiple files than scp IMHO.

rsync -avzh /path/to/dir/ user@remote:/path/to/remote/dir/

Update

You can use rsync via ssh by adding the '-e' switch:

rsync -avzh -e ssh /path/do/dir/ user@remote:/path/to/remote/dir/

#!/usr/bin/expect -f
spawn scp -r BASE.zip abhishek@192.168.1.115:/tmp
expect "password:"
send "wifinetworks\r"
expect "*\r"
expect "\r"

What about wildcards or multiple files?

scp file1 file2 more-files* user@remote:/some/dir/

rsync is a program that behaves in much the same way that rcp does, but has many more options and uses the rsync remote-update protocol to greatly speed up file transfers when the destination file is being updated.

The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection, using an efficient checksum-search algorithm described in the technical report that accompanies this package.

---

    Copying folder from one location to another

---

   #!/usr/bin/expect -f   
   spawn rsync -a -e ssh username@192.168.1.123:/cool/cool1/* /tmp/cool/   
   expect "password:"   
   send "cool\r"   
   expect "*\r"   
   expect "\r"  

If you are ok with entering your password once for every run of the script, you can do so easily using an SSH master connection.

#!/usr/bin/env bash

USER_AT_HOST="user@host"  # use "$1@$2" here if you like
SSHSOCKET=~/".ssh/$USER_AT_HOST"

# This is the only time you have to enter the password:
# Open master connection:
ssh -M -f -N -o ControlPath="$SSHSOCKET" "$USER_AT_HOST"

# These do not prompt for your password:
scp -o ControlPath="$SSHSOCKET" file1.xy "$USER_AT_HOST":remotefile1.xy
scp -o ControlPath="$SSHSOCKET" file2.xy "$USER_AT_HOST":remotefile2.xy

# You can also use the flag for normal ssh:
ssh -o ControlPath="$SSHSOCKET" "$USER_AT_HOST" "echo hello"
ssh -o ControlPath="$SSHSOCKET" "$USER_AT_HOST" "echo world"

# Close master connection:
ssh -S "$SSHSOCKET" -O exit "$USER_AT_HOST"

You can do it with ssh public/private keys only. Or use putty in which you can set the password. scp doesn't support giving password in command line.

You can find the instructions for public/private keys here: http://www.softpanorama.org/Net/Application_layer/SSH/scp.shtml

There are 2 quick ways of achieving this:

1. Using scp

   #!/usr/bin/env bash
   
   password="YOURPASSWORD"
   username="YOURUSERNAME"
   dir_origin="YOURSOURCEDIRECTORY"
   dir_destination="REMOTEDESTINATION"
   Ip="SERVERIP"
   
   echo "Uploading files to remote server...."
   sshpass -p "$password" scp -rC $dir_origin $username@$Ip:$dir_destination
   echo "File upload to remote server completed! ;)"
   
   

2. Using rsync

   #!/usr/bin/env bash
   
   password="YOURPASSWORD"
   username="YOURUSERNAME"
   dir_origin="YOURSOURCEDIRECTORY"
   dir_destination="REMOTEDESTINATION"
   Ip="SERVERIP"
   
   echo "Uploading files to remote server...."
   sshpass -p "$password" rsync -avzh $dir_origin $username@$Ip:$dir_destination
   echo "File upload to remote server completed! ;)"
   

**NOTE :**You need to install sshpass (eg by running apt install sshpass for deb like os eg Ubuntu) that will enable you to auto upload files without password prompts

This will work:

#!/usr/bin/expect -f

spawn scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no file1 file2 file3 user@host:/path/
expect "password:"
send "xyz123\r"
expect "*\r"
expect "\r"
interact

here's bash code for SCP with a .pem key file. Just save it to a script.sh file then run with 'sh script.sh'

Enjoy

#!/bin/bash
#Error function
function die(){
echo "$1"
exit 1
}

Host=ec2-53-298-45-63.us-west-1.compute.amazonaws.com
User=ubuntu
#Directory at sent destination
SendDirectory=scp
#File to send at host
FileName=filetosend.txt
#Key file
Key=MyKeyFile.pem

echo "Aperture in Process...";

#The code that will send your file scp
scp -i $Key $FileName $User@$Host:$SendDirectory || \
die "@@@@@@@Houston we have problem"

echo "########Aperture Complete#########";

Try lftp

lftp -u $user,$pass sftp://$host << --EOF--

cd $directory

put $srcfile

quit

--EOF--

The command scp can be used like a traditional UNIX cp. SO if you do :

scp -r myDirectory/ mylogin@host:TargetDirectory

will work