While ETL continues to be the predominant data integration technology in use today, the sheer size and complexity of today's information and application systems require new levels of innovation that lower costs, deliver rapid time to value, and put more control into the hands of business users.
A new study by IdealNet, a professional business expertise and technical services company, takes a look at the data integration technology curve, from the 1980s to present day, examining first, second and third generation solutions. It also compares first generation tools such as conventional ETL with newer open source and data virtualization technologies, and includes recommendations as to which technologies would be most appropriate based on an organization's data integration requirements.
The results of the study are captured in a video, titled "An Analysis of Data Integration Technologies." Some of the key findings:
· Informatica is the leading provider of ETL, clearly defines ETL for batch oriented bulk file transfer
· Talend and Pentaho are the open source leaders with associated ETL and BI strategies
· ETL is limited in scalability – each connection is essentially standalone
· Queplix 3rd generation product – leading data virtualization solution, better suited to integrate 2, 3, 4 or more sources
· Unique advanced data virtualization platform provides greater capability, lowers costs
To learn more, see http://bit.ly/qY0RFx.
Tweet This: Data Integration Study: ETL vs. Next Generation Technologies - http://bit.ly/qY0RFx
For parties interested in Queplix, data integration, data management, data quality, data virtualization, data integration software, master data management, MDM, business intelligence, informatica
Thursday, July 28, 2011
Saturday, July 23, 2011
Friday, July 22, 2011
Manual Process and MDM
Cost , Inefficiencies and Lack of Scalability Plague Conventional MDM - http://bit.ly/eOjIlN
Author:
Michael Zuckerman
CMO
Queplix
It is interesting to review all of the market opportunity data relating to master data management. Visible almost immediately is the huge chunk of professional services required for implementation. Upon closer inspection this seems to be a very large percentage of the market. Vendors will tell you, and rightfully so, that all of this is paid for by the cost of a successful project. The return on investment is there. Of course, but this requires a successful and timely completion. However, it is still a large chunk of your cost and manual process also reads risk.
So look closer. There is a closely related but sometimes separately tracked market called (master data management) integration services. That market seems to include an even larger swath of professional services packaged with a few ETL links, extra data storage, and a data mart (or two). In total, when you add this all up, you can see that master data management is mostly about manual processes and human interaction. The software expense feels like, on a very good day, something approximately over 70% of the total expense for initial implementation and likely approaching 80%. The software in that bucket is a small slice of the overall expense. It is all about manual process and people.
If you don't see this expense you are not looking closely enough, IMHO.
Perhaps the costs are, in fact, even higher. My data about expense is all anecdotal. I ask the question and get answers. But most people are not completely sure. The vendor information is somewhat optimistic about implementation time. The industry analysts are a bit more astute, but they tend to leave the integration expense and consulting in another bucket. It is very hard to tell. When all is said and done, no one has likely been able to capture all of the email, conference calls, political wrangling and hallway meetings required to align your data models on a global basis. It is likely far more expensive than you know.
Manual process and human interaction does not scale well. It is not more efficient. It gets less efficient as the number of intermediaries to measure progress and “manage” grow. These resources cost more, raise the overhead and generally slow things down. You have alot of politics to deal with. Politics is the art of getting people to do what you want them to do as a function of influence, not authority. The results are not easily predicted, cooperation is not assured and the goals of the different parties are not in alignment. It should be very clear, especially in the wake of the economic debacle of Q4 2008, that the needs and interests of these groups are focused first on their top goals and objectives. Master data management may not be one of them.
Of course, to make this more difficult, all of this manual process and human interaction involved is distributed far and wide. You are necessarily dealing with people from diverse working groups and these working groups are attached to every line of business, IT team, geographical entity and P&L in your company. The larger you are, the better the return on investment should work for you, the more diverse the interaction you must sponsor. Cooperation is always a function of relationship, and you cannot always build close working relationships over a videoconference line or a conference call. Cooperation does not work well by command, except perhaps in the military. So you need to get on planes and visit your various work groups, at least once or twice. More process and time tied to hours of your time.
Lack of automation is a big problem. Right now everything is wrestled to the ground by committee. Emails and policy statements then flow out regarding data governance decisions. It is all manual process – not much is automated. There is no computer software enforcing and implementing the policies. You may need to scrub the duplicates out and if you do not… nothing happens.
Data governance teams need the same sorts of tools the information security teams have for implementation and enforcement. For example, your password expires and you must change it or you lose access to the system; you cannot install a 3rd party piece of software that is not approved; your email storage is limited in size and in some financial institutions must be auto archived and/or expired depending on the date and regulatory oversight; and so on. Automation needs to support authority but there are no tools, dashboards or automated workflow to tie this to legacy MDM architectures in a meaningful way.
Tweet This: @Queplix Video Discusses Manual Process and MDM - http://bit.ly/eOjIlN
Follow Queplix on Twitter at: http://www.twitter.com/QueplixCorp
Resource library: For more information about data virtualization and application integration, visit Queplix Resource Library - http://bit.ly/fSkNMD
For parties interested in Queplix, MDM, master data management, data virtualization, data integration, data integration software, business intelligence, composite software, informatica, boomi
Author:
Michael Zuckerman
CMO
Queplix
It is interesting to review all of the market opportunity data relating to master data management. Visible almost immediately is the huge chunk of professional services required for implementation. Upon closer inspection this seems to be a very large percentage of the market. Vendors will tell you, and rightfully so, that all of this is paid for by the cost of a successful project. The return on investment is there. Of course, but this requires a successful and timely completion. However, it is still a large chunk of your cost and manual process also reads risk.
So look closer. There is a closely related but sometimes separately tracked market called (master data management) integration services. That market seems to include an even larger swath of professional services packaged with a few ETL links, extra data storage, and a data mart (or two). In total, when you add this all up, you can see that master data management is mostly about manual processes and human interaction. The software expense feels like, on a very good day, something approximately over 70% of the total expense for initial implementation and likely approaching 80%. The software in that bucket is a small slice of the overall expense. It is all about manual process and people.
If you don't see this expense you are not looking closely enough, IMHO.
Perhaps the costs are, in fact, even higher. My data about expense is all anecdotal. I ask the question and get answers. But most people are not completely sure. The vendor information is somewhat optimistic about implementation time. The industry analysts are a bit more astute, but they tend to leave the integration expense and consulting in another bucket. It is very hard to tell. When all is said and done, no one has likely been able to capture all of the email, conference calls, political wrangling and hallway meetings required to align your data models on a global basis. It is likely far more expensive than you know.
Manual process and human interaction does not scale well. It is not more efficient. It gets less efficient as the number of intermediaries to measure progress and “manage” grow. These resources cost more, raise the overhead and generally slow things down. You have alot of politics to deal with. Politics is the art of getting people to do what you want them to do as a function of influence, not authority. The results are not easily predicted, cooperation is not assured and the goals of the different parties are not in alignment. It should be very clear, especially in the wake of the economic debacle of Q4 2008, that the needs and interests of these groups are focused first on their top goals and objectives. Master data management may not be one of them.
Of course, to make this more difficult, all of this manual process and human interaction involved is distributed far and wide. You are necessarily dealing with people from diverse working groups and these working groups are attached to every line of business, IT team, geographical entity and P&L in your company. The larger you are, the better the return on investment should work for you, the more diverse the interaction you must sponsor. Cooperation is always a function of relationship, and you cannot always build close working relationships over a videoconference line or a conference call. Cooperation does not work well by command, except perhaps in the military. So you need to get on planes and visit your various work groups, at least once or twice. More process and time tied to hours of your time.
Lack of automation is a big problem. Right now everything is wrestled to the ground by committee. Emails and policy statements then flow out regarding data governance decisions. It is all manual process – not much is automated. There is no computer software enforcing and implementing the policies. You may need to scrub the duplicates out and if you do not… nothing happens.
Data governance teams need the same sorts of tools the information security teams have for implementation and enforcement. For example, your password expires and you must change it or you lose access to the system; you cannot install a 3rd party piece of software that is not approved; your email storage is limited in size and in some financial institutions must be auto archived and/or expired depending on the date and regulatory oversight; and so on. Automation needs to support authority but there are no tools, dashboards or automated workflow to tie this to legacy MDM architectures in a meaningful way.
Tweet This: @Queplix Video Discusses Manual Process and MDM - http://bit.ly/eOjIlN
Follow Queplix on Twitter at: http://www.twitter.com/QueplixCorp
Resource library: For more information about data virtualization and application integration, visit Queplix Resource Library - http://bit.ly/fSkNMD
For parties interested in Queplix, MDM, master data management, data virtualization, data integration, data integration software, business intelligence, composite software, informatica, boomi
Friday, July 15, 2011
Vision for the Future: The Evolution of the Data Integration Market - http://bit.ly/qY0RFx
One of the major challenges organizations face today is how to bridge applications together in a way that enables true data mobility - integrating critical information across different business functions such as HR, sales, accounting, and customer support, so that all they key components of the organization can be preserved and shared among different application systems to drive greater business value.
A newly released video from Queplix™ called Vision for the Future: the Evolution of the Data Integration Market, provides an overview of first, second, and third generation data integration technologies, as well as the emergence of fourth generation, or Next Generation (NGEN), and the capabilities organizations should expect to see in 2012 and 2013.
The overview points out that, contrary to what most might believe, legacy or first generation ETL (extract, transform, load) technology is still predominantly used for data integration today. Further, many of the early integration cloud vendors are simply hosting 20-year old ETL technology in their one-year old clouds.
In sharp contrast, advanced data virtualization technology is powering third and fourth generation data integration products that represent the next wave. The Next Generation will be driven by a powerful convergence of data integration, management, virtualization, abstraction, persistent servers for automated data quality and governance, cloud and on-premise systems, virtual master data management and more. With NGEN, organizations will achieve true data mobility.
To learn more, see http://bit.ly/oVVzv1.
Tweet This: Evolution of Data Integration - http://bit.ly/oVVzv1
Follow Queplix on Twitter at: http://ping.fm/b2LnG
For parties interested in Queplix, data integration, data management, data quality, data virtualization, data integration software, master data management, business intelligence, informatica
A newly released video from Queplix™ called Vision for the Future: the Evolution of the Data Integration Market, provides an overview of first, second, and third generation data integration technologies, as well as the emergence of fourth generation, or Next Generation (NGEN), and the capabilities organizations should expect to see in 2012 and 2013.
The overview points out that, contrary to what most might believe, legacy or first generation ETL (extract, transform, load) technology is still predominantly used for data integration today. Further, many of the early integration cloud vendors are simply hosting 20-year old ETL technology in their one-year old clouds.
In sharp contrast, advanced data virtualization technology is powering third and fourth generation data integration products that represent the next wave. The Next Generation will be driven by a powerful convergence of data integration, management, virtualization, abstraction, persistent servers for automated data quality and governance, cloud and on-premise systems, virtual master data management and more. With NGEN, organizations will achieve true data mobility.
To learn more, see http://bit.ly/oVVzv1.
Tweet This: Evolution of Data Integration - http://bit.ly/oVVzv1
Follow Queplix on Twitter at: http://ping.fm/b2LnG
For parties interested in Queplix, data integration, data management, data quality, data virtualization, data integration software, master data management, business intelligence, informatica
Wednesday, July 13, 2011
Automating Data Quality: A Template for Data Integration - http://bit.ly/qY0RFx
Keeping data consistent across business functions such as marketing and customer support is essential for the vitality of today's organizations. But when companies need to integrate data between applications, the consistency of data, or data quality, becomes an issue. Basic problems such as null fields, duplicate records, syntax problems, range problems and more, impact data quality and with multiple application sources and master data management, it becomes a complex and expensive problem to solve.
In this video, Michael Zuckerman, chief marketing officer for Queplix™ Corp delivers a faster, easier method to maintaining data quality, reducing risk, and improved results.
To learn more, see http://bit.ly/qY0RFx..
Tweet This: Automating Data Quality - http://bit.ly/qY0RFx
Follow Queplix on Twitter at: http://ping.fm/PoOPY
For parties interested in Queplix, data integration, data management, data quality, data virtualization, data integration software, master data management, business intelligence, informatica
In this video
To learn more, see http://bit.ly/qY0RFx..
Tweet This: Automating Data Quality - http://bit.ly/qY0RFx
Follow Queplix on Twitter at: http://ping.fm/PoOPY
For parties interested in Queplix, data integration, data management, data quality, data virtualization, data integration software, master data management, business intelligence, informatica
Subscribe to:
Posts (Atom)