Follow dba-ninja.com

Subscribe to RSS feed  Follow @jackvamvas - Twitter

*Use the Comments section for questions

dba-ninja.com Links

Dba_db2_button

Sqlserver_dba_button

Using S3 as a DMS Target

26 June,2024 by Rambler

These are the specific steps I completed to get a DMS job to use Amazon s3 as a  DMS target . As well as the mechanics for creating a DMS Migration Job targeting an RDBMS as a source server and S3 as target - you should also consider Performance. 

Multiple factors influence the throughput - for some extra guidance on DMS performance - read -How can I speed up my DMS migration ?

 

For the purposes of these notes - I used the name "sql-to-s3" 

Use the  Amazon S3 console,  create a bucket. 

    -> Created a new bucket "sql-to-s3"
Select the bucket that you created, and then choose Create folder.

    -> Created a new folder "sql-db-target"
Enter a folder name, and then choose Save.

 

Create an IAM policy   (replace "bucket" ) with the bucket name created. This policy forms the basis for the DMS task to interact with the target s3 bucket 

-> Created a new policy called "sql-to-s3"

This policy definition was copied from AWS documentation, check link below in background reading 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:PutObjectTagging"
      ],
      "Resource": [
        "arn:aws:s3:::bucketname/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::bucketname*"
      ]
    }
  ]
}

 

Create a new IAM role & attach the IAM policy 

--> Choose "DMS" as the service

--> Attached the IAM policy "sql-to-s3" to a new IAM role called "sql-to-s3"

--> Make a note of the ARN associated with new IAM role. It will be required for the Target Endpoint configuration

arn:aws:iam::12731263633:role/sql-to-s3

Create your DMS target endpoint to s3 

Open the AWS DMS console, and then choose Endpoints from the navigation pane.
Choose Create endpoint, and then select Target endpoint.

--> Created an endpoint called "sql-to-s3"
Enter the Endpoint identifier, and then choose Amazon S3 as the Target engine.

-->Added "sql-to-s3" and select Amazon s3 as target engine 
Paste the Role ARN that you copied into the Service Access Role ARN field.

--> Added "arn:aws:iam::12731263633:role/sql-to-s3"
Enter a Bucket name and Bucket folder.

--> Added "sql-to-s3" for both 
Under Endpoint settings, add your Endpoint settings, if you have any. For more information, see Endpoint settings when using Amazon S3 as a target for AWS DMS.

--> Executed a connection test and got an error message -

Test Endpoint failed: Application-Status: 1020912, Application-Message: Failed to connect to database.

 

This link has a step by step troubleshooting guide. The troubleshooting will take you through the essential steps required . 

How can I troubleshoot Amazon S3 endpoint connection test failures when using AWS DMS?

for my specific problem - I checked through all the points and finally arrived at the passage related to :

Create VPC endpoints so that your replication instance can access all source and target endpoints that are used by AWS DMS. For more information,

I investigated the VPC endpoint related to the s3 service , checked through the details and identified a number of contradictions related to the access restrictions for the s3 service.

Once I adjsuted the restricted access via the Policy associated to the VPC endpoint , I was then able to create a successfull Test connection from DMS to s3

 

Some background reading 

Can I use an Amazon S3 bucket as an AWS DMS target?

Using security socket layer (SSL) on DMS Endpoints

 

Author: Rambler (http://www.dba-ninja.com)


Share:

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment on Using S3 as a DMS Target


dba-ninja.com