[xquery-talk] ETL for XML !??? Nobody realizes that XML tools even EXIST ....

[Ihe Onwuka]

I have had to implement  workarounds when trying to deploy XSLT/XQuery on
datasets circa 250MB, presumably some hardware jiggery pokery could
increase my processing capacity butt there clearly is a limit.

With plain text files this was simply a matter of running the
transformation on files that had fragmented courtesy of grep, awk and
friends, but the XML input required the  replication of a homespun
mapreduce to get the required result, which to be fair that was a bit more
than a straightforward ETL.

Point being there was some X(SLT|QUERY) jiggery pokery (Bill McLaren RIP)
going on, many would throw in the towel at the first heap error and never
get to the stage I got to. Secondly the bash parts of the pipeline which
include  joins and sorts that aggregate across about 15 files run much much
quicker. Hence  I started with an XSLT transformation that did everything
because it looked like a simple upconversion  and ended up with the XSLT
transformation reduced to a final step that runs on heavily pre-processed
and fragmented files and still takes the bulk of the processing time.

I don't know the extent to which the availability of streaming capabilities
(both for text and XML) would change that scenario but the fact is that
making streaming only available in the enterprise versions of these
products has left a lacuna for other products to step in and fill. That's
not a criticism and it is not my place or intention to opine on the release
strategy of Saxon or any other offering, but it does seem to be a logical
conclusion.

Of course that's not to say that it would make a blind bit of difference to
these people. They are still at circa 1980 on the evolutionary cycle (just
discovering that you actually do need a query language) so remember that
before you remind them it is not 1995.







On Thu, Jun 11, 2015 at 6:41 PM, daniela florescu <dflorescu at me.com> wrote:

> Just look at this discussion on Linkedin:
>
> https://www.linkedin.com/grp/post/3990648-6013602791098953731
>
> Dana
> _______________________________________________
> talk at x-query.com
> http://x-query.com/mailman/listinfo/talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://x-query.com/pipermail/talk/attachments/20150612/84e8a75a/attachment.html>