Showing posts with label ETL. Show all posts
Showing posts with label ETL. Show all posts

Tuesday, April 28, 2009

ETL using Microsoft Excel, XML and XSLT

Microsoft Excel VBA is a powerful tool to provide a Facade for simple ETL process.

  • GUI designed and implemented by Microsoft Excel VBA form
  • Business Logic implemented by XSLT
  • Data is stored as XML format



Sub OpenUserForm()
frmExchangeRate.Show
End Sub

Sub xmltocsv()
' Dim oDOM As MSXML2.DOMDocument
' Dim oXML As MSXML2.DOMDocument
' Dim oXSL As MSXML2.DOMDocument
Dim oDOM As MSXML2.FreeThreadedDOMDocument
Dim oXML As MSXML2.FreeThreadedDOMDocument
Dim oXSL As MSXML2.FreeThreadedDOMDocument
Dim oXSLTemplate As XSLTemplate
Dim xslProc As IXSLProcessor
Dim strHTML As String
Dim strTransform As String
Dim currentDir As String

currentDir = CurDir()
Set oDOM = CreateObject("MSXML2.FreeThreadedDOMDocument")
oDOM.async = False
' oDOM.Load currentDir & "\" & frmExchangeRate.txtXSLFilename.Text
oDOM.loadXML Sheets("Run VBA").Range("A1").Value
Set oXSL = CreateObject("MSXML2.FreeThreadedDOMDocument")
oXSL.async = False
oXSL.Load currentDir & "\" & frmExchangeRate.txtXSLFilename.Text
' oXSL.loadXML Sheets("Run VBA").Range("B1").Value
Set XSLTemplate = CreateObject("MSXML2.XSLTemplate")

XSLTemplate.stylesheet = oXSL
Set xslProc = XSLTemplate.createProcessor()
xslProc.addParameter "GMFile", currentDir & "\" & frmExchangeRate.txtGMFilename.Value
xslProc.addParameter "IBMFile", currentDir & "\" & rmExchangeRate.txtIBMFilename.Value
xslProc.addParameter "ISOFile", currentDir & "\" & frmExchangeRate.txtISOFilename.Value
xslProc.input = oDOM
xslProc.transform
strTransform = xslProc.output
'your XSLT stylesheet should be saved as unicode or UTF not ansii
'note encoding instruction maybe needed for european language encoding say swedish characters
' strTransform = oDOM.transformNode(oXSL)
strHTML = "" & vbCrLf & _
"<root>" & strTransform & "</root>"
WriteFile "ERMSExchangeRate" & Format(Now, "yyyymmddhhmmss") & ".tbl", strTransform
'
'the above XSLT transform with xsl file converts this to a flat csv format file
Set oDOM = Nothing
Set oXML = Nothing
Set oXSL = Nothing
Set XSLTemplate = Nothing
Set xslProc = NothingEnd Sub

Public Sub WriteFile(ByVal sFileName As String, ByVal sContents As String)
' Dump XML String to File for debugging
Dim fhFile As Integer
fhFile = FreeFile
' Debug.Print "Length of string=" & Len(sContents)
Open sFileName For Output As #fhFile
Print #fhFile, sContents;
Close #fhFile
Debug.Print "Out File" & sFileName
End Sub

XSLT File
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:msxsl="urn:schemas-microsoft-com:xslt" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:user="mynamespace" xmlns="urn:schemas-microsoft-com:office:spreadsheet" version="1.0"
>
<msxsl:script implements-prefix="user"><![CDATA[
]]></msxsl:script>
<xsl:output method="text"/>


<xsl:param name="File1"/>
<xsl:param name="File2"/>
<xsl:param name="ISOFile"/>

<xsl:variable name="iso4217_list">
<root> <xsl:for-each select="document($ISOFile)//ss:Workbook/ss:Worksheet[1]/ss:Table[1]/ss:Row"> <item> <country><xsl:value-of select="translate(ss:Cell[3]/ss:Data/text(),'abcdefghijklmnopqrstuvwxyz','ABCDEFGHIJKLMNOPQRSTUVWXYZ')"/></country>
<code><xsl:value-of select="number(ss:Cell[4]/ss:Data/text())"/></code>
</item>
</xsl:for-each>
</root>
</xsl:variable>


<xsl:variable name="gm_exchange_rate_list">
<root>
<xsl:for-each select="document($File1)//ss:Workbook/ss:Worksheet[1]/ss:Table[1]/ss:Row[position() > 1]">
<xsl:variable name="country"><xsl:value-of select="ss:Cell[4]/ss:Data/text()"/></xsl:variable>
<item>
<time_stamp><xsl:value-of select="ss:Cell[1]/ss:Data/text()"/></time_stamp>
<rate_type><xsl:value-of select="ss:Cell[2]/ss:Data/text()"/></rate_type>
<data><xsl:value-of select="ss:Cell[3]/ss:Data/text()"/></data>
<country><xsl:value-of select="$country"/></country>
<country_name><xsl:value-of select="ss:Cell[5]/ss:Data/text()"/></country_name>
<code><xsl:value-of select="msxsl:node-set($iso4217_list)/ss:root/ss:item[ss:country/text()=$country][1]/ss:code/text()"/></code>
<multiplier_rate><xsl:value-of select="ss:Cell[6]/ss:Data/text()"/></multiplier_rate>
</item>
</xsl:for-each>
</root>
</xsl:variable>

<xsl:variable name="eLedger_date"><xsl:value-of select="substring(msxsl:node-set($gm_exchange_rate_list)/ss:root/ss:item[2]/ss:time_stamp/text(), 1, 8)"/></xsl:variable>
<xsl:variable name="eLedger_time"><xsl:value-of select="substring(msxsl:node-set($gm_exchange_rate_list)/ss:root/ss:item[2]/ss:time_stamp/text(), 9, 6)"/></xsl:variable>
<xsl:template match="/">HDR<xsl:text>^</xsl:text>4<xsl:text>^</xsl:text>1<xsl:text>^</xsl:text>R<xsl:text>^</xsl:text>1<xsl:text>^</xsl:text>1<xsl:text>^</xsl:text><xsl:value-of select="$eLedger_date"/><xsl:text>^</xsl:text><xsl:value-of select="$eLedger_time"/><xsl:text>^</xsl:text>General Motors<xsl:text>^</xsl:text>E-Ledger<xsl:text> </xsl:text>
<xsl:for-each select="msxsl:node-set($gm_exchange_rate_list)/ss:root/ss:item">
<xsl:variable name="code"><xsl:value-of select="ss:Cell[1]/ss:Data/text()"/></xsl:variable>
<xsl:value-of select="ss:code/text()"/><xsl:text>^</xsl:text><xsl:value-of select="ss:multiplier_rate/text()"/><xsl:text> </xsl:text>
</xsl:for-each>TRL<xsl:text>^</xsl:text><xsl:value-of select="count(msxsl:node-set($gm_exchange_rate_list)/ss:root/ss:item)"/>
</xsl:template>
</xsl:stylesheet>


Process Steps
1 Save "eFXControllerSettlementsMonthlyBalSheet.xls" as "eFXControllerSettlementsMonthlyBalSheet.xml" ][option: xml spreedsheet (*.xml)] in the same folder
2 Save "ersexrate001.tbl" as "ersexrate001.xml" ][option: xml spreedsheet (*.xml)] in the same folder
3 Open EXCEL_VBA.xls from Microsoft Excel (Do not double click the file, use menu Start->Programs->Microsoft Office->Microsoft Office Excel 2007, and then open the file)
4 After ERMS_VBA.xls file is opened, a dialog box displays. Click "Run" button.
5 Message indicates that a new "tbl" file is created in the same directory.
6 Open the newly created tbl file in Microsoft Excel.

Monday, April 27, 2009

Vendor Service Cost Projection using Excel Macro

Vendor will provide secure data transfer service. The cost is based on the byte size of the files transferred.

ETL Process using Microsoft Excel

ETL project usually involves data cleanup. An architect many times acts as a bridge between Business Analyst and System DBA.

If you are familiar with Design Patterns, to Business Analyst, the architect is like a "Bridge" pattern; to DBA, the architect is like a "Builder" pattern;

Creating Query using Microsoft Excel
Task statements:
Business Analyst has provided (see figure below):
Column A: SchemaName and Table Name
Column B: Field Name
Column C: Type of data (From business point of view, PI is referred to Personal Information)
Column D: Commentys (From business point of view)
Column E: Data Type
Column F: New Value

Business Analyst wants to replace all PI data with the value of '0'. The list is not completed yet, but it will have more than 100 rows.
System DBA asks Architect to provide executable Oracle SQL scripts similar to the text in Column G.



Architect Solution: Use Excel functions to create the command which can be easily copy / paste to each row.

The Command in Column G:
=CONCATENATE("UPDATE TABLE ", REPLACE(INDIRECT(ADDRESS(ROW(), COLUMN()-6)),FIND("_", INDIRECT(ADDRESS(ROW(), COLUMN()-6))), 1,"."), " SET ", INDIRECT(ADDRESS(ROW(), COLUMN()-5)), " = ", INDIRECT(ADDRESS(ROW(), COLUMN()-1)), ";")

Excel Functions:
CONCATENATE()
In Excel, the Concatenate function allows you to join 2 or more strings together.
The syntax for the Concatenate function is:
Concatenate( text1, text2, ... text_n )
There can be up to 30 strings that are joined together.
Applies To:
Excel 2007, Excel 2003, Excel XP, Excel 2000

REPLACE()
In Excel, the Replace function replaces a sequence of characters in a string with another set of characters.
The syntax for the Replace function is:
Replace( old_text, start, number_of_chars, new_text )
old_text is the original string value.
start is the position in old_text to begin replacing characters.
number_of_chars is the number of characters to replace in old_text.
new_text is the replacement set of characters
Applies To:
Excel 2007, Excel 2003, Excel XP, Excel 2000

INDIRECT()
In Excel, the Indirect function returns the reference to a cell based on its string representation.
The syntax for the Indirect function is:
Indirect( string_reference, ref_style )
string_reference is a textual representation of a cell reference.
ref_style is optional. It is either a TRUE or FALSE value. TRUE indicates that string_reference will be interpreted as an A1-style reference. FALSE indicates that string_reference will be interpreted as an R1C1-style reference. If this parameter is omitted, the Indirect function will interpret string_reference as an A1-style.
Applies To:
Excel 2007, Excel 2003, Excel XP, Excel 2000

ADDRESS()
In Excel, the Address function returns a text representation of a cell address.
The syntax for the Address function is:
Address( row, column, ref_type, ref_style, sheet_name )
row is the row number to use in the cell address.
column is the column number to use in the cell address.
ref_type is optional. It is the type of reference to use. It can be any of the following values:
ValueExplanation
1Absolute referencing.For example: $A$1
2Absolute row; relative column.For example: $A1
3Relative row; absolute column.For example: A$1
4Relative referencing.For example: A1

If this parameter is omitted, the Address function assumes that the ref_type is set to 1.
ref_style is optional. It is the reference style to use: either A1 or R1C1. It can be any of the following values:
ValueExplanation
TRUEA1 style referencing
FALSER1C1 style referencing


Explanation
If this parameter is omitted, the Address function assumes that the ref_style is set to TRUE.
sheet_name is optional. It is the name of the sheet to use in the cell address. If this parameter is omitted, then no sheet name is used in the cell address.
Applies To:
Excel 2007, Excel 2003, Excel XP, Excel 2000

ROW()
In Excel, the Row function returns the row number of a cell reference.
The syntax for the Row function is:
Row( reference )
reference is optional. It is a reference to a cell or range of cells.
Note:
If the reference parameter is omitted, then the Row function assumes that the reference is the cell address in which the Row function has been entered in.

For example, the Row function used in the picture above returns 1 because the Row function has been entered in cell A1. Therefore, it assumes the following formula:
=Row(A1)
Applies To:
Excel 2007, Excel 2003, Excel XP, Excel 2000

COLUMN()
In Excel, the Column function returns the column number of a cell reference.
The syntax for the Column function is:
Column( reference )
reference is optional. It is a reference to a cell or range of cells.
Note:
If the reference parameter is omitted, then the Column function assumes that the reference is the cell address in which the Column function has been entered in.
For example, the Column function used in the picture above returns 1 because the Column function has been entered in cell A1. Therefore, it assumes the following formula:
=Column(A1)
Applies To:
Excel 2007, Excel 2003, Excel XP, Excel 2000

FIND()
In Excel, the Find function returns the location of a substring in a string. The search is case-sensitive.
The syntax for the Find function is:
Find( text1, text2, start_position )
text1 is the substring to search for in text2.
text2 is the string to search.
start_position is the position in text2 where the search will start. The first position is 1.
Note:
If the Find function does not find a match, it will return a #VALUE! error.
Applies To:
Excel 2007, Excel 2003, Excel XP, Excel 2000