KEY FEATURES AND CAPABILITIES OF MATILLION ETL FOR AMAZON REDSHIFT

MATILLION ETL FOR AMAZON REDSHIFT (AVAILABLE ON THE AWS MARKETPLACE)​

  • Matillion ETL for Amazon Redshift is a cloud-based ETL (Extract, Transform, Load) tool that is specifically designed to work with Amazon Redshift. Amazon Redshift is a popular data warehousing solution that allows users to store and analyze large amounts of data in a scalable and cost-effective way.​
  • allows users to easily integrate data from various sources, transform the data into a format that is suitable for analysis, and load it into Amazon Redshift. ​


HERE ARE SOME KEY FEATURES AND CAPABILITIES OF MATILLION ETL FOR AMAZON REDSHIFT:​

  1. CLOUD-BASED: MATILLION ETL FOR AMAZON REDSHIFT IS A FULLY CLOUD-BASED SOLUTION, WHICH MEANS THERE IS NO NEED TO INSTALL ANY SOFTWARE ON YOUR LOCAL MACHINE. YOU CAN ACCESS THE PLATFORM FROM ANYWHERE WITH AN INTERNET CONNECTION.​
  2. PRE-BUILT CONNECTORS: MATILLION ETL FOR AMAZON REDSHIFT PROVIDES PRE-BUILT CONNECTORS FOR A WIDE RANGE OF DATA SOURCES, INCLUDING DATABASES, CLOUD APPLICATIONS, AND FILES. THIS MAKES IT EASY TO INTEGRATE DATA FROM VARIOUS SOURCES INTO AMAZON REDSHIFT.​
  3. POWERFUL TRANSFORMATIONS: MATILLION ETL FOR AMAZON REDSHIFT PROVIDES A POWERFUL AND INTUITIVE INTERFACE FOR TRANSFORMING DATA. YOU CAN EASILY PERFORM COMPLEX TRANSFORMATIONS SUCH AS AGGREGATIONS, JOINS, AND PIVOTS USING A DRAG-AND-DROP INTERFACE.
  4. SCALABLE: MATILLION ETL FOR AMAZON REDSHIFT IS DESIGNED TO BE HIGHLY SCALABLE, ALLOWING YOU TO PROCESS LARGE VOLUMES OF DATA QUICKLY AND EFFICIENTLY.​
  5. REAL-TIME DATA INTEGRATION: MATILLION ETL FOR AMAZON REDSHIFT SUPPORTS REAL-TIME DATA INTEGRATION, WHICH MEANS YOU CAN EXTRACT AND LOAD DATA IN REAL-TIME OR NEAR REAL-TIME, DEPENDING ON YOUR REQUIREMENTS.​
  6. DATA ORCHESTRATION: MATILLION ETL FOR AMAZON REDSHIFT PROVIDES A RANGE OF DATA ORCHESTRATION FEATURES, INCLUDING SCHEDULING, JOB MONITORING, AND ERROR HANDLING. THIS MAKES IT EASY TO MANAGE YOUR DATA INTEGRATION WORKFLOWS AND ENSURE THEY RUN SMOOTHLY.​ ​

OVERALL, MATILLION ETL FOR AMAZON REDSHIFT IS A POWERFUL AND FLEXIBLE SOLUTION THAT ALLOWS USERS TO EASILY INTEGRATE, TRANSFORM, AND LOAD DATA INTO AMAZON REDSHIFT. ITS PRE-BUILT CONNECTORS, POWERFUL TRANSFORMATIONS, AND REAL-TIME DATA INTEGRATION CAPABILITIES MAKE IT A POPULAR CHOICE FOR ORGANIZATIONS LOOKING TO STREAMLINE THEIR DATA INTEGRATION WORKFLOWS AND GAIN DEEPER INSIGHTS FROM THEIR DATA.

​

MATILLION ETL FOR AMAZON REDSHIFT LAUNCHES ON THE AWS MARKETPLACE

​
​Matillion ETL for Amazon Redshift is an AMI delivered ETL/EL-T tool which allows users of Amazon Redshift to transform data quickly and easily within the Redshift platform.​

It allows users to unlock significant performance gains in Amazon Redshift by giving you the ability to process millions of rows of data, in a matter of seconds.​

  • Unlock Redshift’s power by processing millions of rows in a matter of seconds.​
  • Modern, beautiful, browser-based environment for full-featured graphical job development.​
  • Fast setup – Launch the AMI and be developing ETL jobs within minutes.​


WHY CHOOSE MATILLION ETL FOR AMAZON REDSHIFT?​

  • Users of Amazon Redshift will be all too familiar with the difficulties involved in getting their data into Redshift, let alone ensuring it is in the right shape. This complicated process often involves having to join, aggregate and de-normalise large data sets which can often be a frustrating and time-consuming task.​
  • The solution of course, is to use an ETL tool to help Extract, Transform and Load data into Redshift. However, traditional ETL tools can be fundamentally slow when it comes to handling larger data volumes.​
  • As a result, many users opt for a different approach, loading their data into Redshift BEFORE transforming it within Redshift itself. This approach is called ‘ELT’ – Extract, Load (into Redshift) then, Transform.​
  • The problem with this approach is that is often requires you to manually write lots of complex code and continuously maintain these scripts. This can be a complicated, slow and very expensive approach, which can drain technical resource from other areas of the business.​


GETTING STARTED WITH MATILLION ETL FOR AMAZON REDSHIFT IS RELATIVELY STRAIGHTFORWARD. HERE ARE THE BASIC STEPS YOU NEED TO FOLLOW:​

  • Create an Amazon Redshift cluster: Before you can start using Matillion ETL for Amazon Redshift, you need to create an Amazon Redshift cluster. This can be done through the Amazon Redshift console.​
  • Launch Matillion ETL: Once you have an Amazon Redshift cluster, you can launch Matillion ETL for Amazon Redshift. This can be done through the AWS Marketplace.​
  • Configure Matillion ETL: When you first launch Matillion ETL, you will need to configure it. This includes setting up your credentials, connecting to your Amazon Redshift cluster, and selecting the appropriate region.​
  • Create a new project: Once Matillion ETL is configured, you can create a new project. This project will be used to manage your data integration workflows.​
  • Connect to your data sources: After you have created a new project, you can connect to your data sources. Matillion ETL provides pre-built connectors for a wide range of data sources, including databases, cloud applications, and files.​
  • Build your data integration workflows: Once you have connected to your data sources, you can start building your data integration workflows. This includes defining your data transformations, scheduling your workflows, and monitoring the progress of your jobs.​
  • Test and deploy your workflows: Once you have built your data integration workflows, you can test them to ensure they are working as expected. Once you are satisfied with your workflows, you can deploy them to your production environment.​


MATILLION ETL FOR SNOWFLAKE ON GCP: SUPPORTING A MULTI-CLOUD STRATEGY​

In response to unprecedented market demand for more cloud options, Snowflake has launched its data warehouse on Google Cloud Platform (GCP). Matillion is thrilled to announce that alongside Snowflake, we are launching Matillion ETL for Snowflake on GCP. ​

Matillion’s award-winning data transformation solutions are purpose-built to work with Snowflake on all major cloud platforms–GCP, Amazon Web Services, and Microsoft Azure – bringing our customers never before realized simplicity, speed, scale, and savings. 


MATILLION ETL SUPPORTS SNOWFLAKE ON GCP​

  • In addition to cloud-native functionality out of the box, Matillion ETL supports multi-cloud architectures. For several years, organizations have maintained a hybrid model of on-premises databases and cloud solutions.​​
  • However, as more companies adopt a multi-cloud strategy, they are retiring on-premises databases and replacing them with cloud alternatives. And using different cloud warehouses and platforms together increases flexibility and introduces cost-efficiencies into data workflows. Matillion ETL can be a critical component of a successful multi-cloud strategy.​


LEVERAGE BOTH SNOWFLAKE AND GOOGLE SERVICES​

  • Matillion ETL for Snowflake on GCP employs the same great technology as other Matillion products, so users can really get the full advantage of cloud computing technology. Users can access all their data sources using more than 80 pre-built connectors or our universal API connector. Simple code-optional transformation components can be combined to create complex transformations to change that data into meaningful information for decision making. ​


FROM WITHIN MATILLION ETL FOR SNOWFLAKE ON GCP, USERS CAN TAKE ADVANTAGE OF A MIX OF SNOWFLAKE AND GOOGLE SERVICES. FOR EXAMPLE, A USER CAN: ​
 ​

  • Build responsiveness into their data loading jobs by scaling up their Snowflake warehouse to run a large query, and then scaling back down with Snowflake’s ‘Alter Warehouse’ functionality–all within an automated Matillion orchestration job.​
  • Monitor ETL job success with alerts and notifications using Google’s native Pub/Sub messaging service, ensuring that business critical data is where they need it when they need it.  ​
  • Use Google Cloud Functions to command-and-control Matillion ETL via its comprehensive API to run event-triggered jobs in response to events inside of GCP, such as an object being created on Google Cloud Storage.​
  • Access data in any cloud storage area (Amazon S3, Azure Blob, Google Cloud Storage bucket) using Snowflake External Stages and Matillion’s flexible SQL component, which enables a multi-cloud strategy by closing data silos.​


MATILLION ETL FOR GOOGLE BIGQUERY (AVAILABLE ON THE GOOGLE CLOUD MARKETPLACE)​

  • Matillion ETL for Google BigQuery is a powerful cloud-based tool that allows users to extract, transform, and load data into Google BigQuery. It is available on the Google Cloud Marketplace and is designed to simplify and streamline the ETL process.​


GET THE MOST OUT OF GOOGLE BIGQUERY​

  • Google BigQuery is a powerful cloud-based data warehouse that allows users to store and query massive amounts of data.​


TO GET THE MOST OUT OF GOOGLE BIGQUERY, HERE ARE SOME TIPS:​

  • Organize your data: Before uploading data to BigQuery, organize it in a way that makes it easy to access and analyze. This may involve partitioning data by date, region, or another key attribute.​
  • Use nested and repeated fields: BigQuery supports nested and repeated fields, which can help to simplify data modeling and make queries more efficient. Use these features where appropriate to optimize your data schema.​
  • Take advantage of caching: BigQuery automatically caches query results, so if you frequently run the same queries, they will run faster over time. Make sure to take advantage of this feature to optimize query performance.​
  • Use standard SQL: BigQuery supports both standard SQL and legacy SQL. Standard SQL is more powerful and easier to use, so make sure to use it whenever possible.​
  • Optimize queries: Write efficient queries by minimizing the amount of data scanned and using filters and aggregations where appropriate. BigQuery provides tools for analyzing query performance, so use them to identify bottlenecks and optimize your queries.​
  • Consider using BigQuery ML: BigQuery ML allows users to build machine learning models directly in BigQuery using SQL. Consider using this feature if you need to perform predictive analytics on your data.​
  • Monitor and manage costs: BigQuery charges based on the amount of data processed, so monitor your usage and take steps to manage costs. This may involve optimizing your queries, setting query quotas, or using cost controls.​


LAUNCHING MATILLION ETL FOR BIGQUERY - GCP​

Overview​

Matillion ETL for BigQuery is available through the Google Cloud Launcher service, hosted on the Google Cloud Platform Marketplace. It's capable of using select Google Cloud Platform services, such as Storage and BigQuery. If you have/t already done so, refer to the documentation on GCP Account Setup for BigQuery and Storage for more information.​


Prerequisites​

Before launching a Matillion ETL instance you will require:​

  • Adequate knowledge about the cloud service account (AWS, Azure, GCP) and Cloud Data Warehouse (Snowflake, Redshift, Google BigQuery and Delta Lake on Databricks) you want to launch.​
  • A user with the correct permissions who can access the intended cloud service account. For more information, read GCP Roles for Launching Matillion ETL.​
  • Access to a cloud storage bucket (S3, Azure, Blob Storage or Google Cloud Storage) to house the transient staging files Matillion used to load data to the cloud.​
  • A network path to access the intended data sources. This may involve working with your network team to enable access to on-premise databases.

​

LAUNCHING MATILLION ETL ON GCP​

  1. Use the following steps to launch Matillion ETL for BigQuery on the Google Cloud Platform:​
  2. Access the Google Cloud Platform Marketplace. Use the magnifying glass icon on the top-right of the Marketplace homepage to search for the product. Type in to the search field "Matillion" or "Matillion ETL" to view the Matillion ETL products. Additional information will also be available about the product, such as an Overview, Pricing, Documentation and Support. For more information on pricing information, read Pricing Guide.​ Click Launch. A new page will open that lets you configure your initial setup for your Matillion ETL instance​.
  3. Complete the following information for your deployment:​
  • Deployment Name​
  • Zone​
  • Machine Type​
  • Disk Type​
  • Dish Size in GB​
  • Network Name​
  • Subnet Name​
  • Firewall rules and tags to allow specific network traffic from the internet​

4. When all the information has been completed, click Deploy at the bottom of the page to launch your Matillion ETL instance.​

5. After deploying, Google Cloud Launcher will begin the process of setting up the instance. In this transitional period, all details for the instance are likely to be listed as Pending. This process typically takes a minute or two.​

6. Once successfully deployed, the following list will change to show completed values. Before leaving this page, make note of the Admin user and Admin password (Temporary) for your Matillion ETL instance because these will be required for the initial login. Your instance can now be accessed by using the link next to Site Address.​

  • Site Address​
  • Admin User​
  • Adin Password (Temporary)​
  • Instance​
  • Instance Zone​
  • Instance Machine Type​

7. To revisit this page, log in to your Google Cloud Platform console, and select Deployment Manager under the Tools heading in the top-left menu of the homepage.​

8. Log in to your Matillion ETL instance using "gcp-user" and use the default password given to you by Google when you launched the instance in your GCP console earlier.​


MATILLION ETL FOR MICROSOFT AZURE​

  • Matillion ETL for Microsoft Azure is a cloud-based ETL tool that enables users to extract, transform, and load data into Microsoft Azure services such as Azure Blob Storage, Azure Data Lake Storage, and Azure Synapse Analytics.​
  • Matillion ETL features a user-friendly visual interface for building ETL workflows, which allows for drag-and-drop components and eliminates the need for manual coding. This makes it easy for users to build and manage data pipelines without the need for extensive technical knowledge.​
  • The tool includes pre-built connectors for popular data sources such as Salesforce, Amazon S3, and Google Analytics, allowing users to easily integrate data from these sources into Microsoft Azure. Additionally, Matillion ETL supports advanced data transformation functions and filters, making it easy to manipulate data within Azure.​


LAUNCHING MATILLION ETL FROM AZURE MARKETPLACE
​Overview​

  • This article describes how to launch and connect to Matillion ETL from the Azure Marketplace.​
  • Matillion ETL provides users with data loading and transformation solutions built for the cloud, for Snowflake, Azure Synapse Analytics, and Delta Lake on Databricks. Matillion ETL takes full advantage of the cloud to process data, and make it useful.​


Prerequisites​

  • Before launching a Matillion ETL instance you will need to register for a Matillion Hub account. You will also require:​
  • Adequate knowledge about the cloud service account (AWS, Azure, GCP) and Cloud Data Warehouse (Snowflake, Redshift or Google BigQuery) you want to launch.​
  • A user with admin permissions who can access the intended cloud service account.​
  • Access to a cloud storage bucket (S3, Azure, Blob Storage or Google Cloud Storage) to house the transient staging files Matillion used to load data to the cloud.​
  • A network path to access the intended data sources. This may involve working with your network team to enable access to on-premise databases.​