Tapdata Replicator Documentation

Tapdata Introduction

Tapdata Replicator is a hybrid database replication product. It is designed to provide real time database change replication between different type of databases. For example, you can replicate data from Oracle to MongoDB, or from MySQL to Elastic Search etc. In addition Tapdata can also be used to replicate data between two MongoDB clusters.

When Do I Need It

Oracle to MongoDB data migraiton
You are building apps on MongoDB, but the data your app needs, such as customer data or order data, are currently in Oracle database. For a number of reasons, you cannot decommission your Oracle just yet. So you use Tapdata to replicate the data from Oracle to MongoDB, in real time fashion.

This also applies to MSSQL, MySQL etc.

MongoDB to MongoDB intra-cluster replication
MongoDB has great built-in replication feature, but it works within a replicaset. If you wish to replicate changes from one cluster to another, or intend to build a dual-active type of deployment, you may use Tapdata to help you to achieve the goal.

Supported Database Types

Supported source databases(Tested versions):

  • Oracle 11g, 12c
  • MSSQL 2012
  • MySQL 5.6, 5.7
  • Sybase ASE 15, 16
  • MongoDB 3.2, 3.4, 3.6, 4.0

    Supported Target databases

  • MongoDB 3.2, 3.4, 3.6, 4.0
  • Elastic Search(To be released)0

  • Install Tapdata

    Choose the correct edition

    Tapdata provides following editions:

      • Tapdata Cloud Free
      • Tapdata Cloud Premium
      • Tapdata Enterprise

    The key difference is between cloud version and enterprise version is where the Tapdata management portal is installed and managed.With the cloud versions, Tapdata team manages the management portal in the cloud. You do not need to install or manage the Tapdata application software. Instead, you only need to download an agent application and install it in your data center(or on your cloud server).With the Enterprise edition, you download and install the whole suite of the Tapdata product locally in your data center.

     

    Tapdata Product Comparison (Version 1)

    Feature Cloud Free Cloud Premium(*) Enterprise
    Installation Method Agent Only Agent Only Entire Product
    Management Portal In the Cloud In the Cloud Local
    Number of Jobs 3 Unlimited Unlimited
    Oracle MongoDB Replication Yes Yes Yes
    MySQL MongoDB Replication Yes Yes Yes
    MSSQL MongoDB Replication Yes Yes Yes
    MongoDB MongoDB Replication Yes Yes Yes
    Sybase MongoDB Replication Yes Yes
    DB2 MongoDB Replication * *
    Job Monitoring Yes Yes Yes
    Agent Management Yes Yes
    User Management Yes
    API Designer Yes
    API Security Yes
    API Gateway Yes
    Online Support Yes Yes
    Phone Support Yes

    * Not yet released.

    * Currently all Cloud Premium features are available to Cloud Free users.

     

    Tapdata Cloud Agent Installation

    1. Prerequisites

    • Chrome BrowserTapdata is currently only tested on Google Chrome browser. We recommend version 51.0.2704 or above.
    • Java 1.8Run the command java -version whether the server has Jdk_1.8 installed.If Java is not installed, you may refer to this page to install Java: Java 8+

    2. Download and Install the agent software

    • Log onto http://cloud.tapdata.io
    • Click Agent Download link on the left navigation menu
    • Download and decompress
        tar -xvf tapdata-agent-x.x.x-xxxx.tar.gz
        cd tapdata-x.x.x
      
    • Open “application.yml” to enter the access code, which is printed on the agent download page.

    3. Start the agent

    • Start the agent
        ./agent.sh start
      
    • Verify agent is running properly, by going to Agents page. You should be able to see your agents listed.
    • Now you can go to http://cloud.tapdata.io to create connections and replication jobs.

    Note the agent must be able to connect to http://cloud.tapdata.io.

     

    Tapdata Enterprise Installation

    1. Prerequisite

    • MongoDB 3.6

      Run the command mongod --version server with MongoDB 3.6 or above installed Please confirm MongoDB architecture, it should be ReplicationSet. Note MongoDB binaries must be in the command search path. You may install MongoDB by following instructions here: MongoDB 3.6+

    • Java 1.8

      Run the command java -version whether the server has Jdk_1.8 installed.

      If Java is not installed, you may refer to this page to install Java: Java 8+

    2. Request Tapdata Enterprise

    Please contact us at team@tapdata.io to request for an Enterprise Edition download link.

    3. Download and Install

    Once you have got the link, download the software and decompress it:

    tar -xvf tapdata-x.x.x-xxxx.tar.gz cd tapdata-x.x.x

    4. Run the Tapdata Enterprise

    Under the installation directory, issue following command:

    ./tapdata.sh start

    If this is first time running, the installation program will ask you to enter the MongoDB connection information.

    Now you may access the Tapdata management portal in your browser. Using following address:

    http://server_hostname:3030

    to access Tapdata. Replace server_hostname with the actual hostname or IP address of the server where Tapdata is installed.

    Login use default admin user account:

    Username: admin@admin.com Password: admin

    Job runtime log prompt: Stopped sync lag maker;

    This error message indicates that there is insufficient permissions to read Oracle redo log. Please refer to Oracle Configuration Guide page to properly setup the access permission with your DBA’s assistance.

    I’m getting this error when running job: java.sql.SQLRecoverableException: No more data to read from socket

    Possible remediations:

  • Check Oracle’s SQLNET.EXPIRE_TIME setting, try to increase it
  • Check system’s ulimit setting
  • Check Oracle connection pool to see if it’s overrun.
  • Check the number of connections on Oracle database
  • Does Tapdata support views?

    In order to provide real time replication capability, Tapdata needs to monitor the transaction log file. Only changes made to physical tables will generate transaction logs. For this reason, views are not currently supported by Tapdata.

    How to change the default user’s login(admin@admin.com)?

    Under tapdata install directory, run following script to chagne the default user login

    ./user.sh