IBM Brian

IBMBrian website transition – I am retiring from IBM at the end of March 2024, so I can no longer be called IBMBrian nor will I have access to the most up-to-date information.  This website may be a “frozen in time” resource or it will undergo changes.  I am working with Amin and his Data Is Everything blog, and he may take over maintenance of this website.  Before I go, I leave you with this…

DataStage-aaS Anywhere; put your execution engine near your data!  DataStage as-a-Service Anywhere supports execution at any location (on any cloud and on premises). The remote engine manifests as a container, so it can be deployed anywhere a container can run, using any container management platform (Docker, Kubernetes, Podman, etc.).  In other words, DSaaS Anywhere does not require Red Hat Openshift to run.

——————-

Data Fabric!  What is it?  Although originated from pontificators and competitors, it’s an architecture instead of a product.  Basically, it’s the design of getting data from data producers to data consumers. IBM has possessed these functionalities for years.

What is IBM’s Data Fabric solution?  It’s various services/functionalities now pre-integrated on a containerized platform called Cloud Pak for Data (CP4D).

What’s the nerdier answer?!?  It is essentially re-platformed Information Server (IIS) plus other technologies, built cloud-native, deployed on Red Hat OpenShift, which can be provisioned as software or as an appliance behind your own firewall, or as Software as a Service (SaaS), or any option in-between!

MORE!!!  Data Fabric technically consists of whichever combination of these services you wish to deploy for your business requirements: DataStage (i.e., ETL) + IBM Knowledge Catalog (IKC; = an understand/trust/use catalog based on legacy IIS Business Glossary + Information Governance Catalog + Information Analyzer + old WKC) + Watson Query (data virtualization) + Data Refinery (data munging/wrangling); other Data Fabric use cases would add Change Data Capture (data replication), Match 360 (master data management entity matching), and the data science services Watson Studio, Machine Learning, and OpenScale (model development, deployment, monitoring, and governance).  See what I mean by pre-integrated?  When you deploy another CP4D capability/service, the integration is already there, and it’s all on the same User Interface.

Although I’ve been involved with DataStage & InfoServer since 1999, I couldn’t possibly be more giddy since Next Generation DataStage went public in 2021.  New innovations are constantly trickling into production, and these features will make these products relevant for decades.  Would you prefer to hear from unbiased experts like Gartner?  See these quotes in the Multicloud data integration ebook:

• “The data integration tool market is resurging as new requirements for hybrid and intercloud integration, active metadata and augmented data management force a rethink of existing practices.”
• “The data integration tool market is resurging as new requirements force a rethink of existing practices.”

Multicloud Data Integration is the new buzzword for “connecting trusted data to the right people”.  Duh; DataStage.  HAVE YOU SEEN NEXT GEN DATASTAGE?  It has the look-and-feel of the legacy Designer we know and love, plus so many more fancies being constantly added.  Click both the links in this paragraph to do your own free labs!

My new calling is to help current and potential Information Server and DataStage (IIS/DS) customers get migrated to Next Gen DataStage and CP4D on DataStage as a Service or as software on Cloud Pak for Data (CP4D).  Why?

• CP4D is a new kind of data and analytics platform with built-in governance
• CP4D simplifies and unifies how you collect, organize and analyze data to accelerate the value of data science and AI
• CP4D is a pre-integrated set of capabilities, delivered in a prepackaged manner, with multiple deployment options
• CP4D’s environment is a modern, cloud-native architecture with built-in agility and resiliency, supporting a portable multicloud containerized data platform
• IBM offers a dual-entitlement path to “modernize” your IIS/DS deployments at your own pace

Ask your IBM sales representative about the IIS/DS Cartridge for CP4D.  The best CP4D feature so far for DataStage has got to be its automatic engine scaling, previously only available in Grid deployments!

THERE’S MORE!!!  Evaluate DataStage as a Service (aka DataStage Next Generation) for free!  Go to the IBM DataStage home page and give it a try.  Even though DataStage Next Gen has completely rebuilt the DataStage design-time Designer as cloud-native, it “feels” like good ol’ DataStage!  And given the shared-nothing architecture of the DataStage Parallel Engine (PX; Parallel Extender), it easily ported to the cloud-native containerized architecture.  If concerned about the migration effort, the intent is to make the upgrade as seamless as possible, supporting nearly all legacy code.  If the stress of maintaining a DataStage environment has become overwhelming, then DataStage as a Service is for you.

[Putting on my Billy Mays hat] BUT WAIT THAT’S NOT ALL!  Customers that have the IIS or DS Cartridge for CP4D will get entitlement for MettleCI!  See how DataStage can now support your CI/CD initiatives, and automate your migration efforts.

MORE!!!  In August 2022, IBM acquired Databand.ai, a “data observability” product.  Databand evaluates historical pipeline outcomes, compares it to incoming workflows, and determines in real-time if there are any anomalies and alerts the appropriate people.  Currently for DataStage, much pre-and-post-processing effort is required to determine if the incoming data conforms to expected parameters and quality, or if totals add up, etc.  That’s not a challenge for near-legendary coders like me, but it should help with future development as Databand integration with NextGen DataStage is on the near-term roadmap!

Anything else?  Actually, yes.  Manta’s data lineage is now available on CP4D!  It’s the coolest lineage tool ever, led by our friend Ernie Ostic, and quite an upgrade to what we used to have.

You’re done, right?  Well, DataStage and CP4D and other IBM Data & AI services can now be deployed on the AWS Marketplace!  Search for “aws marketplace ibm” if my link doesn’t work for you.

Have you heard of the term “DataOps”?  Think of it as DevOps for data; efforts to streamline the delivery of business-ready data.  Thus, the “Organize” pillar in CP4D…in turn, the functions of Information Server…which means that InfoServer does most of the work under the covers for CP4D!  It’s the most exciting, modern pillar.  Proof: the “Collect” pillar (like databases, file systems, etc.) traces its roots to ancient times, as does the “Analyze” pillar (data science, business intelligence, etc.) when philosophers attempted to describe the process of human thinking as the mechanical manipulation of symbols.  Yes, there are those that wish to convert human behavior into 0’s and 1’s, but many still prefer to be provided with trustworthy business-ready data and use their own brains!

Speaking of history, DataStage 1.0 first shipped on on January 21, 1997.  See these articles about DataStage’s past, present, and future.  Thank you Dennis James for selling the first copy of DataStage!  Dennis still maintains the de-facto DataStage help reference called DSXchange!

If I had lawyers, I’m sure they’d make me say this…

This website is about IBM DataOps from my point of view.  (As such, there is the usual disclaimer that the postings on this site are my own and aren’t necessarily representative of IBM’s positions, strategies or opinions.)

This is also where you can find help with DataStage or another Information Server tool.  It’s a brain dump of information I’ve accumulated since being introduced to DataStage in 1999.  Hopefully it’s not a small website.

Also:

• Posts about Hints and Help for debugging, best practices and tips
• Links to Resources and Support
• Posts about IBM Software information
• A Calendar page for upcoming IBM and KC DataOps / DataStage / Information Server / CP4D User Group events

Best of all, a Search can quickly find the single post you seek, as opposed to scrolling through long webpages.

Bonus: the website is easily viewed from a desktop, laptop, tablet or phone…perhaps even from that big screen TV in your second home in the Caribbean.