Server Migration Tool
    • Dark
      Light

    Server Migration Tool

    • Dark
      Light

    Article Summary

    Warning: The Server Migration Tool and this page are deprecated and exist to aid customers on older versions of Matillion ETL. The Server Migration Tool is now upgraded and integrated into the Matillion ETL client. See this article for more information.

    Note: Your network administrator must ensure that the target Matillion ETL server allows incoming TCP/IP connections from the source Matillion ETL server.

    The server migration tool allows users to push all assets from current server to new server. This tool runs as a shell script when placed on the server you wish to move. Through the tool, users are able to filter what is migrated, although everything will be migrated by default.

    This can be useful in a number of cases:

    • Setting up a new live environment by pushing all assets from a staging environment
    • Setting up a test environment by pushing a single project from live to a test environment to diagnose
    • Migrating a Single Node Matillion Setup to a Cluster by provisioning a new cluster using Cloudformation and then pushing everything to it from the existing environment.

    To use this tool, simply copy the attached scripts to your server (the Matillion ETL instance), SSH onto the server and run the python script with the target hostname (including port if required) as an argument. It may be necessary to update your python installation before running the tool.


    It is also required that API permissions are granted to the user for both source and target server.

    The tool comes with help that can be accessed through the --help argument. A list of optional arguments is given below.

    ArgumentDescription
    --source-server SOURCE_SERVERSource Server Hostname (Default: http://127.0.0.1)
    --skip-driversSkip JDBC Driver migration (true/false)
    --skip-profilesSkip API Profile migration (true/false)
    --skip-projectsSkip Matillion Project migration (true/false)
    --skip-schedulesSkip Schedule migration (otherwise schedules are migrated but disabled) (true/false)
    --skip-sqssetupSkip SQS Listener setup (even when included queues are initially disabled) (true/false)
    --skip-oauthSkip OAuth Token migration (true/false)
    --allow-deletesAllow deleting same-named resources. Projects are deleted and recreated. (true/false)
    --project-filter PROJECT_FILTERRegular Expression to match against Group- name/Project-name/Version-name; default .* (migrate everything)

    Their usage is as below:

    migrate_server.py [-h] [--source-server SOURCE_SERVER] [--skip-drivers][--skip-profiles] [--skip-projects][--skip-schedules] [--skip-sqssetup] [--skip-oauth][--allow-deletes] [--project-filter PROJECT_FILTER] target_server


    Example

    Below is a generic example for running the migration tool from the EC2 Console with the necessary updates that may be required to run to script. Simply download migrate.py to /tmp and use the following 3 commands (replacing the source and target server IP with your own).

    Below is a generic example for running the migration tool from an SSH session with the necessary updates that may be required to run to script. Simply download migrate.py to /tmp and use the following 3 commands (replacing the source and target server IP with your own).

    Note: If running the Migration Tool from the source or the target server then use your private-ip or 127.0.0.1:8080 to identify localhost instead of using the IP address. Servers may not recognise their public ip and may throw an error.

    • sudo yum install python34-pip.noarch
    • sudo pip-3.4 install requests
    • sudo /tmp/migrate_server.py --allow-deletes --source-server http://source-server-ip http://target-server-ip
    • You will be asked to provide source credentials (ec2-user/instance-id) and target credentials (ec2-user/instance-id). The script will then proceed to migrate all available assets.

    Note: Schedules on the new server will be disabled by default.

    • Disable SQS integration on the old server and shutdown the server.
    • On the new server, enable SQS Integration and Enable all schedules.
    Attachments